Minimum Viable Station Documentation: Recipes

Last month I created and posted about a Minimum Viable Station document. There was such an overwhelming and positive response to this doc! I’m really so happy to see it expand into such a wealth of expert advice. However, I don’t want newbies to feel overwhelmed at the size of this document. I want anybody, at any level, to be able to set up their own digitization system within the constraints of their financial situation, space, and time. Also, this existing doc doesn’t get down into the nitty-gritty details of how much a setup would cost (which is going to vary dramatically, even when confined to ‘recipes’, as you’ll se below), which is important when planning any project (especially when you have to ask other people to spend money, or have to raise it yourself).

So I made a recipes document!

Minimum Viable Station round two! Let’s do this! Does your institution have a setup you’d recommend to others, or have you done this work for grant-seeking or internal-financing purposes? Please contribute your specs! Do you see something wrong? Please leave a comment in (either of) the document(s)!

2016 Reflection and 2017 Goals: Retrospective and refactoring

Welcome to the Ashley Blewer annual report! It’s that time of year again, where I start thinking about where I was at this time last year and where I am now. Here were my goals for 2016.

2016

So. What did I do?

Working

Two projects I worked on shipped: LaMaMa Digital Collections and Screen Slate. Honestly I ended up doing basically zero work on Screen Slate other than attend meetings and complain about things and fretting but not actually write code. However, so happy to see its success. And so happy for LaMaMa’s success as well!

My job @nypl continues to be diverse and exciting, working on a myriad of applications both existing and new, public and non-public, in multiple languages and parts of the tech stack, for patrons and for staff. I am happiest this way. I continued work on MediaConch and QCTools signalserver. MediaConch has been a big part of my life for the past 3 years and it’s strange to realize that this project will come to an end later this year.

Conferences

I gave a talk at Code4lib with Dinah Handel called Free your workflows (and the rest will follow): community-driven AV solutions through open source workflow development. I also did a LOT of work organizing and ensuring a successful conference video streaming experience for everyone. I do not think I will do that again, but I am glad that I did it.

I gave a talk at iPRES 2016, Status of CELLAR: Update from an IETF Working Group for Matroska and FFV1 based on a whitepaper written by Dave Rice and me, available here.

I gave a talk (without being there) at Northeast Historic Film with Travis Wagner. Thanks, Travis, for dealing with the a/v headache of playing a video of my portions. The talk was entitled Buying Lobster Rolls With Linden Dollars: Virtual Tourism and Digital Visions of New England in Second Life and Beyond.

For AMIA, I didn’t talk this year but helped with organizing two streams, Digital Preservation and DIY Community Archiving. I will also probably not do this again, but I am again glad that I did it.

Finally, I (along with Dave Rice and Jerome Martinez) organized a symposium called No Time to Wait!: Standardizing FFV1 and Matroska for Preservation. All the videos and slides are here. I never want to do this again but also will probably do this again.

I also taught THREE workshops: at iPRES, at Tate, and at NYU. Phew!

Travel (not for work)

I went to Disney World for my birthday with some top-notch pals. I spent a week in France with another top-notch pal. I took a weekend trip to my home state to see other top-notch pals and get a friends-tattoo! I was afraid of using all of my vacation days for other-work-obligations, but I did not do that! Also, several pals came to visit me! Friends are important and I’m happy to have so many great ones, near and far.

Misc.

  • Had a chapter in a book published along with amazing-person Jess Rudder.

  • Got this baller-ass tattoo.

  • Wrote about Beyonce and video formats three times. Here’s one and here’s another.

  • (Hopefully) empowered others to learn and contribute through ffmprovisr, open-workflows, Minimum Viable Station and other community efforts. I don’t know if this was a stated goal, but it’s 100% always one of my biggest goals: lower barriers, empower others, facilitate growth and knowledge-sharing.

  • Made a few OSS commits in addition to the above!

  • Regarding “personal projects”, I didn’t make enough time for this as I wanted to but I did a couple of one-off projects like Barthes Tarot or media-id-posters. I made a lot of progress on an on-going, intense art project that I hope to totally wrap/ship in 2017. I made some progress on other yet-unreleased projects that deserve priority in 2017. I refreshed my personal website and this blog’s layout (style update, refactoring of codebase), which makes me feel better and lighter.

  • There’s also this.

What did I want to do?

I wanted to take on less new projects and I wanted to work less (or work less on compensated things and more on personal projects). I did not do these things, but I also knew I wouldn’t do these things. I joined a gym and went regularly, but then I quit going regularly. Now I am going again. But this is like a 50% success rate for goal accomplishment.

Some terrible things happened to me in 2015, although didn’t really start to cause damage until 2016. I do not want to carry these ghosts with me into 2017, but I’m afraid that is not totally preventable. I cut myself some slack for the time lost dealing with these things, but didn’t manage to not feel guilty for not working all the time.

2017

What I said last year still rings true: I have two problems: I care a lot and I care about a lot of things.

Overall, I don’t think much has changed. I suppose the theme for 2017 is “refactoring.” I’m beginning to get nervous. I am afraid of becoming lazy, becoming complacent, of taking things for granted, for getting too comfortable. Although when I started drafting this post, I felt like I hadn’t accomplished anything in 2016 but now I see that I actually did a lot, and did many of the things that I set out to do!

Anyway, hello 2017. I will be succinct and vague.

  • Be less lazy.
    Trust me when I say that I am actually secretly a monster.
  • Increase vigilance.
    The political climate was already lousy and it’s about to get wild. It’s important to be prepared.
  • Minimize.
    I’m clearing out things I do not use and not buying anything new that is non-essential. This isn’t really different from how I normally live my life by default, so maybe it’s not a goal other than padding for the first two points and motivation to not buy new pants if I don’t need them.
  • Get organized.
    My Evernote notes are a mess, and so are many other things!
  • Less wasteful.
    Eat better, drink less, cook more, pack lunches. Don’t buy things due to lazy behaviors. Save that money and give it to people/organizations that really need it.
  • Watch more movies, read more books.
    This is important.
  • Take more vacation than in 2016, not less.
    This is also important.
  • More programming!
    I’m still not very good at this and I want to be better. I did a lot of projects this year but I want a non-work project that challenges me thoroughly (like, for longer than a day) in this way.

Minimum Viable Station Documentation

Hello, internet of people!

Something I’ve been thinking about for a long time and have started working on is the creation of documentation for people who want to set up their own analog audiovisual media digitization stations (called elsewhere as memory labs, personal archiving labs, transfer stations, et al). Aside: I’m also super influenced by my goal of supporting an in-the-works Southeast U.S. Transfer Station by any (remote) means possible. 2017 goals!

I’m specifically interested in this documentation’s ability to facilitate realistic fiscal expectations so budgets can be adequately prepared for before getting started (or for the purposes of asking for funding from an organization or using crowdsourcing methods).

Here’s the thing, though… I need your help and expertise. I am not an expert in this, I do not do this full-time (or even significantly part-time). And even if I were an expert – it’s bullshit for any person to think that they know more than many people working together. So, I ask you.

Here is the document.

This is the goal: Provide what the options are, then give a recommendation. If possible, give a tiered recommendation like “$$$ / $$ / $” for “if ya got a bit of cash” to “if you need a bare-bones minimal setup.” Note that a minimal setup does not correlate with “archival quality” standard but rather a “getting this stuff off of an obsolete format before it is no longer possible to realistically or inexpensively to do so.” We don’t have time for that shit.

Here is the document again!

P.S., N.B.: Don’t let best practices block progress.

P.P.S.: Part two – recipes!

FACING / FEAR / FIRST

Note well: I wrote this over a year ago and forgot to post it. Here it is, right in time for it to not be near graduation-time whatsoever. But also here it is, a preparing-you-for-your-job-hunt pep talk on the internet for whenever you need it.

Hello.

This is a much-fuller version of professional talk that I didn’t give in anticipation of the slightly nauseated-anticipation-feeling I know is felt deep within people exiting their final semester and entering the world again.

Some guidelines for prospering. 🖖

1 Pretend to be Beyonce!

This is also known as “fake it ’til you make it.” Confidence is important to have, though. Not arrogance — but a clear understanding that you are a smart and capable person. Know yourself (and know your limits). It’s hard to see how far you’ve come when you are still in the middle of it.

2 Trust your mentors!

Part 2 – in case the first part is too hard right now because you are wallowing in job application rejection emails (or, even worse, the silence).

First, mentorship is incredibly important. I am forever grateful to Dave Rice, Heather Heckman, Mark Cooper, and my Flatiron School peers for pushing me forward and helping me get where I want to be. I couldn’t have done it without them. But getting a good mentor is primarily luck-based. You happen to be working with or meet a kind and good person that sees potential in you that you don’t yet see for yourself. You can push your luck a little bit though, so I want to talk about what you can do to help facilitate these kinds of relationships and the things I encourage my own mentees to do.

2.5 Just say yes!

If you have a good mentor and you also work with them, they are your boss or whatever, and they want you to do something but you are scared and unsure that you even know what they are talking about, that you are even capable of understanding what they are talking about — just say yes. Just say yes. I mean, if they are your boss but they are also a jerk, don’t say yes, but I am talking about people you love and respect. Just say yes. This usually goes unstated but it was told to me once and you can repeat it in your head until it sticks: If your boss asks you to do something and you totally mess it up and can’t do it and everything is terrible — it’s not YOUR fault. It’s THEIR fault. It’s on them. They are your manager and its their responsibility to ensure you perform optimally. That’s why they are there, not to make you suffer and laugh at you when you Hypothetically or Maybe Literally Ruin Everything. So if they are giving you an opportunity to do something, it’s because they have faith in you and know you can succeed in doing it. I cannot stress this enough. And you really need to have the same confidence in yourself. We frequently don’t. Rise to the occasion. That’s the only way you get better.

3 Keep learning!

Don’t stop! I spend a lot of time feeling like an idiot, but feeling like an idiot is how you know you are learning.

Here is my guide/hierarchy of learning when you are on your own:

  1. I wanna learn this. Good, that’s some good motivation there, it’s good you want to learn something and are excited about something. But sometimes just wanting to do something isn’t enough. Especially when there are SO many cute cats on the internet.

  2. I thought of a project and I’m gonna learn this. This helps me a lot, it helps that I can find a concrete project and fulfill it and I will feel like I have definitely learned That Thing.

  3. I told someone about this project so now I have to learn this (y’know then it’s a little embarrassing when someone asks you and you are like “oh yeah it’s going, um, I spent 17 hours on Facebook this week). Embarrassment.

  4. I am doing this project for someone so I have to learn this. Now someone else is involved to really hold you accountable, because it’s a thing that they want. Your friend, your mom, a colleague, whatever. You have something to deliver to them and that really helps motivation. Guilt.

  5. I am getting paid to do this so I have to do this. Especially if they give you deadlines. Deadlines are so great. Me giving away this secret is a little embarrassing especially if anyone who has ever paid me to do anything is currently reading this and I know that they are ugh agh ack– but in the end it doesn’t really matter because I’ve always been able to deliver the results they are looking for, right? How I got so brash, I don’t know. But, yeah: Have faith in yourself, quit saying no out of fear, learn a lot.

We can transcend all of this and move into Top Level Learning: LEARN WITH OTHER PEOPLE. If you are in a group of people learning one thing and you are the asshole dragging behind or not showing up, you are not gonna feel great and that feeling (wanting to avoid that feeling) will motivate you to move forward.

Caveat: There are good reasons to say “no” to something. Most of these reasons have to do with time/effort/energy. If you’re already busy, don’t agree to things that are going to take up too much time for you to tackle successfully.

Caveat 2: The thing about agreeing to do something you don’t really know how to do is that you might spend 4 hours figuring out what the hell you are doing and then 4 hours actually doing the thing. So keep that in mind when you are budgeting your time. Working all the time isn’t fun and it’ll make you sad. So know what your thresholds are and expect to work double-hard at something if you are learning first, because it means you have to invest in yourself first. But in the end it’s totally worth it because you are investing that time in yourself, and when the project is done, you’ll be much better the next time.

I spend a lot of my time like 🙀 cat-screaming-in-fear-emoji 🙀, but it’s because I’m constantly pushing myself to learn, and learning is actually hard and scary and it hurts a lot. I encourage you to embrace the cat screaming in fear part of yourself and not be afraid to say yes to opportunities, because that’s the best way to become a better version of yourself.

4 Collaboration not competition!

Seriously so tired of this bullshit that I don’t even want to go into it. Just don’t fucking do it, just be fucking nice. There is nothing more boring than people who are still worried about other people. What a waste of time. It will bring you down. Cling to your peers because you are all in this together, now and far into the future. Especially if you are part of a cohort, it can be hard to deal with other people getting jobs before you get a job: your self-esteem drops and you worry about getting left behind. But be happy when they get jobs and know that yours is coming next.

5 Don’t give up!

This is an undervalued, underpaid field and there are more people that want to be in it than are in it. The job hunt takes time. You probably won’t land your dream job right away. You might have to take two part-time jobs. You might have to take a job that isn’t in the field. If you are in-field, you’ll probably be an underfunded, grant-funded employee. Stick with it and you’ll get there. I’m not saying “everything is going to be okay.” I’m saying “you have to keep working your ass off if you want to get what you want.”

I applied for at least 60 jobs when I graduated with an MLIS and heard nothing and spent a lot of time feeling so sad. You should pretend to be Beyonce during this nightmare moment but know that in order to truly become Queen Bae, you have to work your ass off for years and years. But remember Queen Bae’s words of wisdom: “You can do everything right and still lose.” Just get up and try again. Be scrappy, fight hard (but not with your peers).

Bonus 1. Stop acting like a student.

Take this as a polite nudging. The sooner you quit acting like a student, the better. I mean, okay, you are a student, but you don’t need to introduce yourself with these “Well, I am only a student…” vibes. Soon you won’t be a student. You’ll just be you. Wait, who are you again? Oh, that’s right, you are Beyonce.

Bonus 2: Write code (maybe)

If you are still getting super nervous, learn how to write some code and jobs slowly but surely start falling out of the sky. Also don’t be upset if writing code just isn’t something you are into, because that’s fine. But at least consider learning as much as you can about some technical concepts. All archivists are digital archivists. If you truly want to live in the dark ages, take a sword-fighting class before you graduate because you will have to murder a tenured academic librarian to claim your spot in the illuminated manuscripts Illuminati. For the rest of us, y’know, there’s database structures and data transformation scripts.

Parsing XML into a CSV with Ruby

So a short while ago, my friend Lorena posed this question…

And thought “ya, I know how to do that and I also have a secret agenda to get all archivists to learn Ruby.” I sent her some pseudo-code but it’s better to follow up in a blog post that actually explains how things work (also, that pseudo-code would not have run).

If you want to play along at home, here is the small ruby script: https://gist.github.com/ablwr/aad01782214cec1632b65bc42559d4ca

And here is the sample xml: https://gist.github.com/ablwr/850638dc5c6a62aef6e1c9bb140bf107

I’ll walk through the lines and explain them accordingly.

First we have to set some things up.

require 'csv'
require 'nokogiri'

Ruby has extra libraries and we will need two of them. One is the CSV library, which comes with Ruby. The other is the Nokogiri library, which can be hard to install but it lets you manipulate XML easily (including HTML). This is telling the script to grab things we need so we can do it the easy way.

A lot of computers already have Ruby installed. Nokogiri takes extra steps (and sometimes will blow up in your face). So this script won’t work until you install Nokogiri. This might be a total nightmare but it also might be totally fine. If you send me error messages, I can send you my secret girlscout techniques to fix it.

To run a Ruby script, you can type ruby /path/to/script.rb in your command line interface of choice.

Next, we need to get our XML and get it in shape.

xml = File.read('hello.xml')
doc = Nokogiri::XML(xml)

I am running this script in my root directory (cd ~ gets you there on a Mac) and the XML file is also saved there. If I downloaded this .rb file and hello.xml and they saved to my Downloads directory, I could still run this script from my root directory, but run it by changing the File.read to File.read(‘Downloads/hello.xml’).

So we read the document but then we have to turn it into a special Nokogiri document so we can easily grab what we need. That’s what the second line is doing, and later we can work on grabbing info out of the variable we just named “doc”.

all_the_things = []

Scoping is important in programming languages, including Ruby. This makes a variable in advance, and the brackets mean it is an empty array. When we fill up this all_the_things later, it will be ready to go.

doc.xpath('//file').each do |file|
  title          = file.xpath("./title").first.text
  filename       = file.xpath("./name").first.text
  identifier     = file.xpath("./identifier/*[contains(text(), 'My display ID')]").text
  secret         = file.xpath("./identifier/secret").attr('secret').text
  all_the_things << [title, filename, identifier, secret]
end

This chunky bit of code is where we go through our first loop using the Nokogiri library…

We have our doc and we know that it is a <document> full of many <file>s, so this first part – doc.xpath('//file').each do |file| is where we start off our loop and grab (using xpath) every XML node that is a <file>. Then we start grabbing the things that we need. We can grab a bunch of stuff, like the title, the file name, and some identifiers. Nokogiri builds on Xpath statements which are good for all archivists to know, since archivists are obsessed with XML. These are saying “give me the first <title>”, “give me the first <name>”, “give me the identifiers that has the phrase My display ID in it” (this is an example of having to grab data out of poorly-formed XML – who would do this?!), and “give me the secret attribute within the secret identifier”. You can see how this is all arranged in the sample XML.

CSV.open('new_file.csv', 'wb' ) do |row|
  row << ['title', 'filename', 'identifier', 'secret']
  all_the_things.each do |data|
    row << data
  end
end

This will save in the directory you ran the script in. Which, like I mentioned earlier, will only work if you are in the same directory as the hello.xml, unless you change the above file path.

The 'wb' flag is added to let Ruby know that we are going to be writing to the file, not just opening it, and that we want to make sure it opens in binary mode (although running it without the b flag is probably fine).

So this is our second loop and it has another loop inside of it. It’s OK to feel dizzy. When you open a CSV, it’s ready to work row by row until you’ve had enough and tell it to stop. First I ask it to make a row with column titles, which is what row << ['title', 'filename', 'identifier', 'secret'] is all about. The << is “shoveling” these strings of text into the first row. I kinda think of it as having a Mary Poppins bag that looks like this [] and then you can just << to add things in there but it pretty much stays the same size because it’s whatever you named it. Ours is named all_the_things.

So I take our all_the_things and I know all of our XML data has been living there for a minute, floating around in magical Mary Poppins Computer Bagspace, so I tell it to take every array inside the array and make one row for every array. Sorry – Nested arrays are confusing so that might need to sink in for a minute. The array looks like this: [["The first file", "one.mkv", "My display ID: 1", "sesame"], ["The second file", "two.mkv", "My display ID: 3", "benne"], ["The third file", "three.mkv", "My display ID: 3", "goma"]]. Everything is lined up perfectly and ready to become CSV rows.

Anyway, we close up our two loops and that’s…

The end! That is it! Assuming you didn’t give up while your computer objected to installing the Nokogiri gem, you should have a beautiful little CSV (if CSVs can ever be considered beautiful – archivists are out of control).

BYE

Here are some bonus resources!