I’d like to measure the levels of harmful gasses in my living room, I can do this with a VOC sensor (volatile organic compound; will pick up CO2, cigarette smoke, burning candles, etc.).

So, I’m going to wire up an ESP32 with a solid state gas detector, and a VOC sensor!

This one here, a ZP07-v4.0 outputs levels low, mid, high of VOCs in the air around it. This will not tell me what the VOCs are though.

This little beauty will though, and give me “measurable” levels of CO, nitrogen dioxide and ammonia. Sweet.

Now, to wire this all together to an ESP32!

Some coding…

Voila, an air quality sensor for my living room, that reports back to my home automation server every 5s. I’ll start graphing the levels soon, see if I can glean any useful info, how many cigarettes smoked, how “stinky” apartment is ?

Woop! I survived the shitshow that was 2020. Now, what to do?

I know, I’ll extend my home IoT network, add more sensors, relays, LEDs, some well-being monitoring schtuff. Planning to control/monitor with HomeAssistant running on an RPi 4 (MQTT server, and ESPHome server).

For the rest of January I’ll be adding an ESP a day!

First, a simple 12V relay, powered by this cute lil ESP-01s. Relay will power 2 computer case fans, 1 which will pull air through a carbon filter; 1 will cool an RPi cluster.

What? Why?

Found acrylic panel in DIY store, was given a vacuum cleaner free if I did “something cool with it”. I kinda wanted access to a vacuum former, but fuck it, might as well build.

Planning

Found 50*50cm 5mm thick sheets for 20 monies, got two. 2m outer edge, bought 2m of nice thick wood, cubic wood too. Found this amazing drill attachment for making big holes of different sizes, bought it.

Sketching

It’s a box, I didn’t bother.

Getting into it!

• 2 x 50cm * 50cm Acrylic sheets 5mm thick
• A drill with 3mm drilly bit
• TIME
• Wood (for frame)
• Silicone sealant
• Tensioning rods
• Heating element
• Hinges
• Clips
• An assortment of screws
• Permanent markers, fine tipped, 2 colours minimum

First things first. Sheets are 50*50cm, I need to.make a frame of outer size 50*50, and inside, a frame of smaller 6mm square wood to provide additional support. In one of the sides, there needs to be a hole for the vacuum cleaner inlet.

With the frame made, and additional supports installed for the acrylic base, cut to the inner dimensions of the frame; I think this will make for a better way to maintain a seal, and thus partial vacuum.

Now to paint! I don’t want to be regarding wood all the time. I use an acrylic plasticote paint, yellow, on inside and out. Hmmm. I should have sanded this frame inside and out first :/ oh well

Now, to put in the base acrylic layer, this will be screwed to the support, and silicon sealant used above and below!

Screwed in, and sealed, the bottom looks like this:

Sealing gaps on the outside was done, and I moved to the top side.

Before laying the top plate, a zigzaged a metric shittonne of sealant, aligned, lowered the acrylic plate, and pushed hard on edges, and started screwing with 6-8mm legnth screws, 5 each side (because science, lol, na I didn’t have enough screws for more ?). Thanks to my trusy Bosch IXO for making the screwing easy (also, omg, it charges over usb!).

Now, all that is left is to test. I have a spare vacuum cleaner ~1800 watts and a hole in the side of this box that almost fits the business-end tube in, would perfectly fit a plumbung tube, I would do a screw adaptor… (I’ll probably slightly enlarged hole by excessive sanding).

NOTE: Testing comes in next post, I do wonder if I need to drill some more teenie tiny 1 mm holes to the top.

NOTE 2: heatingy bit comes in next buildlog!

Introduction

At EF23 I wanted to go to see a talk given by XX on analysis performed on e621’s posts and tags, I didn’t make it (hungover I think).  Berf, oh well, I read a  little in the EF newspaper on the talk, and forgot until I refound the newspaper clearing office. That motivated me to take a little look at e621 scraping, and what I can learn from the tags. Bonus: i’m learning to create interactive data analysis dashboards with python, this could be a neat little project!

Scraping e621

I need to get the post data associated with each image posted to e621, thankfully an API is exposed by e621 to facilitate automated (bulk) harvesting of their data. A scraper was written to acquire the post data, respecting server load concerns (a 200 posts per 10s rate limit was imposed upon the scraper), and left running… Over the course of 2 months, 500,000 posts were ingested and their metadata stored in a mySQL database, image files were not downloaded. A failure rate of approximately 2% was accepted (encoding errors for weird characters, post description overfilling the database field length of 3000 characters, etc.) as it was deemed the remaining 98% of posts will provide a sufficiently representative sampling of the whole.

A graph of the number of posts, arranged by year and month is presented below:

Interesting, this tells us a couple of things at first glance: there are some busy periods for posting (spikes), and there’s a general upward trend in the number of posts per month (fandom becomes more popular / more art is produced / more art is posted to the site as it becomes more popular).

Now we have the posts ingested, time to look at some other stuff. What are the most popular tags used on e621? I’ll query the top 15, and collect the rest of the data as “other”, and plot as a pie chart. Top 15 tags, Other presented with their percentage occurence in the legend.

Cool. Lets look at how the top 5 of these tags change with time. “Anthro” follows the shape of the post vs. time curve, and is basically a defining tag present on each post. The others show more muted variation following loosely the shape of the posts v time plot, with some subtly changes; for example male and female switch in relative popularity at a couple of points. This could warrant further investigation.

Hmmm… Lets take a look at the species present in the e621 posts. Thankfully e621’s metadata differentiates between species, copyright, character, and descriptive tags.

Closing statement

OK, that’s enough for a quick look at the e621 data. I’ll continue looking deeper, have some near ideas, and will make these interactive plots available too ^.^

Info

I’ve been playing with the server instance this site is hosted on after a few behind the scenes changes were made by host. WordPress will be kept for the moment as a blogging tool; more excitingly, django (python webserver) has been installed, along with mediaWiki, and a nicer gallery app.

Django will be used for serving some cute little visualisations of the fandom’s twitter behaviour i’m working on. MediaWiki will be used for build logs of other projects (fursuit head, and the gloves (see twitter)).

stay tuned….

meaw

Over the past few weeks I made some improvements to the twitter mining code (see last post, on building a dataset of media for CV training), and let it “run in the wild”. Over the course of a few weeks it harvested ~2.5 million tweets, and ~1 million images. These tweets and images come from ~5000 users, predominantly furries (seemingly biased towards fursuiters), which have been identified by the program (publicly accessible information only was collected; hashtags and bio keywords used as primary filter, some manual manipulation of the users was performed to ensure purity of collected dataset).

This is already a fairly substantial dataset, sufficient in size to perform some quite complex analysis on. Using, for example, the location of a tweet and hashtags contained within it, one could plot what is regionally trending around the globe.

One could take this a step further, by performing sentiment analysis on the text contained within the tweet, to gauge the fandom’s opinion on the hashtag (topic). Extending, with some insight into the context of a hashtag, if for example it is associated with a convention, deep insights into the convention could be harvested: overall “happiness” of tweets mentioning it, “happiness” of attendees during the convention, “happiness” of foreign attendees, estimates of attendees from regions… this could be a fun and useful thing to do!

textblob (python package) was used to perform the rudimentary linguistic analysis. Sentiment analysis on tweets using textblob.blob.sentiment.polarity. Concurrently, some additional textblob functions were used; language, word.lemmatize, and sentence.tags: to determine sentence language (deal with those damn bi-lingualists!); to stem the word, for later frequency analysis; Point of Speech (POS) tagging, for better text analysis (dealing with self-identifying terms in Twitter Bios is a good example; but also harvesting “negative/positive words” used to describe conventions). Results from this process are stored in additional tables in the database. As this runs as a “worker” service, it processes tweets with a lag, currently ~20% of 2.5million tweets have been text processed.

Current stats:

492,953 tweets have been processed, containing 7, 307,107 words, of which 398,229 are unique (across 5 primary languages: english, japanese, french, spanish, german)

onto looking at hashtags….

hashtags, while scraped and added to tweet db, were not an initial consideration… so no specific tables were created for them, to expedite analysis. Time to build another worker, and some more tables. As data already collected is being processed (hashtags exist as a json string in the tweet db, thus the string needs to be read, split and added to a secondary db, with some extra information, to facilitate quicker queries later).  This worker was launched a day go, and has processed ~200,000 tweets, containing 85,000 distinct hashtags; enough to have some kind of insight, although likely to suffer from sample biasing, due to the initial method of building the tweet db (populating by scraping tags: #furries, #fursuit, #fursuitfriday), which is evident from the top 5 results.

The top 14 hashtags (so far…) are:

Convention related tags start appearing at 17th position in frequency list, with FurFest, MWFF, AnthroCon being the first 3 to appear.

I expect these frequencies for the hashtags to change drastically as completion is approached for the tweets processing.  Currently (as the location feature is not used across all tweets, and sample biasing), there are poor statistics for large portions of the globe, with massive areas without any data, I expect this to change with time. Time statistics are quite sporadic, hashtags clustered in particular months, a consequence of incomplete dataset processing

Sentiment (average) has not been reported for these hashtags for a few reasons, mainly because I feel it will be too unrepresentative, as, soon the hashtag processing worker will overtake the tweet text processing worker (as that is much slower) which provides the sentiment of the tweet text associated with the hashtag.

Visualisations are being prepared, and will be posted when there are more meaningful statistics for sentiment and location.

ToDo: dashboard for readers to navigate data

Intro

I have to get more heavily into CV, and show some kind of portfolio to help with getting a better job. So I thought “hey, why not make a fursuiter tagger for all those EF pictures and videos you took”. Ticks all the boxes with getting into CV more heavily; nice bit of practice for scraping (training images) and db building too (lol, a graphDB for social network buildng from tagged photos).

Details, concerns, etc.

I’ll defer ethical quandries about dealing with artistic IP by stating I think it’d be polite to ask permission to attempt the identification (naming) of fursuiters in photos; for the more general problem of training a “fursuiter recogniser” publicly accessible photos will be harvested (from multiple sources) and used to train CV algorithms.

So I ask a few furs. TaoruPanda was nice enough to volunteer to participate for identification. Other volunteer are welcomed! Off to get photos… TaoruPanda has a twitter (common among furries) and a FurAffinity account (ubiquitous among furries), both of these will be scraped.

Twitter can be used for building the general training set of images, tags #fursuit, #fursuitfriday (for example) are practically assured to contain a positive image of a fursuiter for this purpose. Thus, twitter shall be scraped for building a training set of images.

Aside: twitter is also interesting due to associated meta data and text found in the tweets. 

Python: Scraping twitter

You will need an account for twitter to programatically access their data (through REST APIs). Further to having an account, you need to enable API access in developer options (this requires associating a phone number with the twitter account, used for a verification code). OAuth used for authentication, makes easier to login with Python, wrappers for the API exist tweepy and python-twitter are nice.

tweepy: nice and easy to use; oversimplified and abstracted some API endpoints, makes trickier for this need.

python-twitter: used in the code below, mainly because natively deals with JSON and exposes full API.

Decide what data is needed:

• Tweeted media from a specific user
• Text, hashtags and metadata for tweeted media
• Tweets without media (?)

This data then needs to be stored for later analysis: media files and dumps of tweets are stored locally (on computer) as images organised in folders, and as JSON dumps respectively; tweet data is stored in databases for quick retrieval, sorting, etc. Two principle database types will be used here, SQL and Graph -type databases. SQL (MYSQL) for flat data storage, graph (Neo4J) for network / topological analysis.

Strictly speaking, for the CV Project, SQL  and tweets with media are all that are required.

<<code to be posted later in entirety>>

Once a sufficiently large collection of images has been made (both with, and without fursuiters) image processing can start. Initially this will involve manual tagging of suiters in images, until a sufficient number of known suiter images is built up to train the CV algorithms.

Stay tuned for next time for: dirty nitty-gritty details, prelim results, problems encountered…. or something like that

TL;DR – LINKS:

Fursuit Parade: https://www.youtube.com/watch?v=E71qcETUu10

80s Dance: https://www.youtube.com/watch?v=6qdiVpRRxak

Post Content:

EF22 is over *sad-face*, epic fun times were had. Now I process and (slowly, the processing takes time) release videos I recorded there (I was the goofball walking around with 6 GoPros mounted to a selfie stick, at dances, con, firepit, parade, etc)

The first full one, of the “fursuit parade” is available (rough) here:

here I say “Rough cut” because there is still some processing to do to make perfect (minor stitching errors are evident in video, and the “virtual horizon” has not been correctly set; both of these need to be manually configured each time the camera rig makes a large movement in the vertical plane, and, well, that happens alot!), for now I hope you’ll be forgiving and put up with slightly weird view/orientation/minor motion-sickness until fixed!

Dance (80s Dance) is being processed by youtube now (first version lacked the correct metadata, and was not processed to 360 rectangular-viewport; instead remains as a orthographic projection, sort-of like world maps, with a sphere flattened to rectangle. Is here is you want to see: https://www.youtube.com/watch?v=6qdiVpRRxak )

I’ll make a post with details on how these were made, and software used. I’m being a little lazy at the moment, and using AVP (AutoPano Video Pro; commercial software from Kolor, now a subsidiary of GoPro), it has the benefit of being quick and semi automated; has the drawback of being too automated at times. PTGui and VideoStitch are beastly (and geeky, more my style 😉 ) but with a backlog of 400Gb of videos, quick first, then proper! I will note that the workflow for PTGui / VideoStitch allows for some truly funky things to be done during the frame-by-frame processing, e.g. calling OpenCV with Python and identifying / tracking objects *grins* (I’ll post on that at some point too)

Meaw for now,

Meawmix

*squee* my con badge! From the lovely Auhrin(?), mohrne.tumblr.com

… it was balls to the wall, but didn’t get to finish the head in time (neighbours complained about Dremel use at night, oops)… will be ready for GLC in November, maybe a regular head too, lol.

Now I’m in Berlin for EF, staying offsite in an apartment XD [blog update,] shower, tea, then TO EF!!!