Convert a MySQL dump to SQLite3

Posted on by david Posted in Uncategorized — No Comments ↓

Sometimes it’s great when you find a tiny tool that does one thing and one thing pretty good. Currently I’m doing a fair bit of back-end development in Django. I usually run with a small sqlite3 database locally instead of having a separate MySQL database. This enables me to quickly copy the file to another computer if needed.

However the production database is a lot more complete than my tiny SQLite database (a lot of people have added content into it). So I needed a way to get all the content into my development database. After some browsing I found a shell script called mysql2sqlite which worked wonders. It takes a mysql dump and converts it into a sqlite3 database file using awk.

https://github.com/dumblob/mysql2sqlite

A keyboard centric view in the browser

Posted on by david Posted in Living in the browser, Web development — No Comments ↓

I’m striving to use the trackpad/mouse as little as possible. The reason is that it will take me longer to reach for the mouse when typing or browsing a website and it’s specific type of movement that can strain your wrist. For many developers and web monkeys this is quite the common knowledge. This blog post is targeted to those who would like to use the mouse and trackpad less. It is also worth mentioning that this is just a short list of  shortcuts that could be found in the browser, but I believe these are the ones I’m using the most frequently.

General
Here are the more general shortcuts which works in most browsers:

  • Go to the address bar (F6)
  • Reload page (CTRL+R or F5)
  • Forced reload (no cache) (SHIFT+CTRL+R)
  • Show page in fullscreen (F11)

Tabs
I constantly have a number of tabs open, by default I place my most used tabs in the top left side of the browser so I can access them quickly with their number.

  • Creating a new tab (CTRL+t)
  • Closing a tab (CTRL+w)
  • Go to next tab (CTRL+PageDown)
  • Go to previous tab (CTRL+PageUp)
  • Open a previously closed tab (CTRL+SHIFT+t)
  • Go to the first tab (CTRL+1)
  • Go to NUM(1,2,3,4,5,6..) tab (CTRL+NUM)

Scrolling and navigating a page
Everyone probably knows page up / page down to scroll on a web page. But they are located in a quite unfortunate place on the keyboard, you can use as easily use the space key.

  • Scroll down a page (SPACE)
  • Scroll up a page (SHIFT+SPACE)
  • Go to top of page (HOME)
  • Go to bottom of page (END)

Zooming
In most browsers you can zoom in text which comes in handy when having a presentation or if the text on the webpage is too small or too big.

  • Zoom in (CTRL + +)
  • Zoom out (CTRL + -)
  • Return to 100% (CTRL + 0)

Editing text in a WYSIWYG editor
In most rich HTML editors,  you can use the normal editing shortcuts that will save a lot of unnecessary movements.

  • Navigate in text (ARROW KEYS)
  • Selecting text (SHIFT+ARROW KEYS)
  • Make selected text bold (CTRL+b)
  • Make selected text italic (CTRL+i)
  • Jump one word forward/backward (CTRL+ARROW KEY) (can also be used when selecting CTRL+SHIFT+ARROW KEY)
  • Go to end of line (END)
  • Go to start of line (HOME)

FFMpeg to the rescue

Posted on by david Posted in Living in the terminal, Video — No Comments ↓

For web publish I needed to downsample several videos, since they were way too large to use in banners. So this post is just for me to remember what I did. I was sure this could be done by the terminal and yes I managed to shrink a 250MB file to 29MB, which was way over my expectations and without any dramatic changes in quality. This of course depends on what kind of video it is. If there are mostly interviews it should be fine.

 

Recording your mistakes

Posted on by david Posted in Living in the terminal, SysOP — No Comments ↓

Do you remeber the driving games in the end of the 90’s early 00’s. They had this fun feature of recording your driving from different angles, so you as the player could review your driving and laugh at how bad/good you were performing. Today I just found out the same for the unix terminal. This can real useful if you are documenting some procedures in a shell. Another use is if you are doing some complicated configuration the first time and later use it for reference next time you doing a similar configuration.

So when you start record something you’ll create a script file and a timing file.

And you can start doing the work you need to record. This also records if you are switching accounts, ssh into another machine and so on, which is real useful. When you are done with recording, either write ‘exit’ or hit CTRL+D.

To replay what you’ve recorded you use the command script replay with the same files you just recorded.

And you’ll see that the script is replaying what you did in the console. The -d option is for speeding up the script by 50%. Otherwise it would replay it in the speed you recorded it in.

 

 

Help I’m stuck in the grub console!

Posted on by david Posted in Living in the terminal, Recovery, SysOP — No Comments ↓

Last day my workstation (which runs Linux Mint 18.2 Sonya) crashed while upgrading Grub2 (the default Linux boot loader). After a hard reset it and I found myself  grub console. After rebooting twice, I realized that something had gone wrong. Grub is one of the things I’ve never bothered to look into, since I’m mostly doing development and not system administration. And experimenting with tenfold of Linux distributions doesn’t appeal to me any longer.

I had quite a lot of work that had to be done, so I reached out for my pack of USB memory sticks and hoped there would be a recovery boot option. After being through 5 different sticks; the only dire option was to trying to boot up the computer via the grub console or reinstalling everything.

First step – orientation – ls and cat

I needed to figure out which one of my harddrive contained my filesystem to boot to. I figured out that the ls command in grub can give a lot of information.

The hd0 stands for harddrive 0, or /dev/sda. The last digit stands for which partition; and the slash in the end means that I’d like to list all the files and directories in the root of that partition. In this case /dev/sda1. After a few adjustments on the harddrive number and the partition number, I finally found my root directory (the volume to boot up on).  Harddrive 1 and partition 2. (hd1, 2). Which also translates into /dev/sdb2 I verified that it indeed was the volume by:

Second – Inform grub about your root filesystem

So I found the root filesystem, which I needed to tell grub about. I declared the root in grub:

I also needed to declare the linux kernel and the initrd image (temporary root filesystem which the system boots upon). They are located in the /boot directory. However, grub2 can work with symlinks and there are usually a symlink to both vmlinuz and initrd.img in the root directory. So I declared these for grub:

The root argument is needed so the kernel knows which one is the root filesystem. In this specific case the disk was the second disk (number 2) and partition 2 which translates to /dev/sdb2.

Third – Boot time

Lastly I crossed my fingers and hope that the machine finally would boot up. By telling grub to boot:

The first time I ran the boot command it didn’t work at all. I’d forgotten the root argument for the kernel. After adding that, the machine booted up and I could finally run grub-update to fix the corrupt grub installation.

Hope it helps anyone who’s experiencing the same issue. It is at least a quick guide for myself for next time :)

 

Force filesystem check on a AWS EC2 (ubuntu 14.04)

Posted on by david Posted in Living in the terminal, SysOP — No Comments ↓

I’ve recently noticed that an Amazon EC2 VPS instance spawn the message when logging in:

So here’s a recipie to solve the issue (as root), if you are in the same situation:

  1. Edit /etc/fstab and locate the line your external storage, the last digit 0 should be changed to a 1 (otherwise it will skip the check of that disk).
  2. Edit /etc/defaults/rcS and change the FSCKFIX=yes (and un-comment if needed)
  3. Create a file called /forcefsck in the root of the server.
  4. Reboot and wait while the server restarts and performs the check.
  5. If solved, revoke the changes you made to /etc/defaults/rcS and /etc/fstab

Hope it’s any help for someone out there.

Help my linux server won’t reboot or halt!

Posted on by david Posted in Living in the terminal, SysOP — No Comments ↓

In some cases it can be impossible to restart your server or just shut it down. The reasons could be many, but most times it concerns I/O blocking that could be caused by a faulty device. If you’re having the server next to you, you’ll just hit the reset button and disconnect the faulty drive. In cases when the server is not in your reach you can send a message to the kernel to HARD reboot. All commands are of course done with a root account.

Keep in mind this is the same as pressing the RESET button on a an active machine. This can corrupt data if unlucky. Dropping 1 into sysrq means that you want to access a system request. This is normally set to 0. And the trigger ‘b’ means reboot system. It’s not good practice, but it helped me to get a system back up again after hung processes eating all the IO.

Removing sensitive details from git history

Posted on by david Posted in Living in the terminal, Programming, Version Control — No Comments ↓

In early stages of software development, it can sometimes be inviting to use a hard coded passwords for quick and dirty access. Later when the project has grown and has a more serious manner of handling authentication, you don’t want old passwords laying around in your git history. The history remains. Warning: Be sure to backup repo before trying this out!

So here is one way to replacing a password string to something else in git. An example: You have been using the password sUp3rs3cr3tpassw0rd since it’s actually a password you elsewhere you want it gone from your git history and replace by HIDDENPASS. This only works on one branch.  So a good idea is to get rid off all other branches and concentrate on the master branch.

This will go through the entire commit history of the branch replacing and rewriting the commit. Depending on the size your project this can take a very long time! It’s not a perfect solution, but it does the job. So lesson learned never use hard coded passwords in code or in a repository. A much better idea is to store it in a text file that never should be included in any commit.

If there’s a file you really want to get rid of in your commit history you can do something like this:

Although, keep in mind that all of these operations is a gateway to a world of merge conflicts for people that are syncing against the repository.

There is also another tool written in Scala that does this a lot faster, and can remove large blobs from the git history called BFG.

Source:
http://www.davidverhasselt.com/git-how-to-remove-your-password-from-a-repository/

BFG:
https://rtyley.github.io/bfg-repo-cleaner/

Search and destroy in bash

Posted on by david Posted in Uncategorized — No Comments ↓

I’ve been doing a lot of web development using the Django framework lately. I’m a big fan of Python and Django since it enables you to quickly get results and keeping the code and file arrangement structure nice and tidy. When I’m trying to figure out what data I’m messing with I’m usually using the Python debugger pdb. When using the django development server and pdb it sometimes causes spawning new server instances when it restarts the server (after code changes) which prevents you to restart the server with CTRL+C.

So today I found a bash phrase which does a ps aux looking for Django instances and stopping them.

This can be applied to many other situations where things needs to be restarted or killed. Thank you Stackoverflow!

Answer on Stackoverflow:
https://stackoverflow.com/a/3510850
Django:
https://www.djangoproject.com/
ipdb:
https://pypi.python.org/pypi/ipdb/0.8.1

GhostScript magic

Posted on by david Posted in Living in the terminal — No Comments ↓

I’m quite used to doing presentation slides in Inkscape. However there’s no proper multi page support in Inkscape. After a quick search I found a person on Stackoverflow that had the same issue. So I found out that you can use GhostScript to merge several PDF’s into one. You can also use it in order to decrease the filesize of a PDF.

Inkscape
https://inkscape.org/en/

Stackoverflow post
http://stackoverflow.com/questions/2507766/merge-convert-multiple-pdf-files-into-one-pdf

 

 

 

Top