Convert multiple files from latin1 to utf8 with vim

If you have several files that need to be converted to utf8, you can do it with vim.

First start vim with:

$ vim $(find . -type f -iname '*.htm*')
Which will find all .htm and .html files from the current directory.

Then in vim, use : to go into command mode.

Run these two commands:
:set nomore
:bufdo set fileencoding=utf8 | w

Reference:
https://stackoverflow.com/questions/4544669/batch-convert-latin-1-files-to-utf-8-using-iconv

Use Curl to read your imap emails

You can use the bash command “curl” to read your imap emails like this:

curl --url "imaps://imapserver.somewhere.net" --user "username:password"

Note: your password will be visible on the command line, so don’t use this on a public server.

So it is actually better to create a file, for instance called secret, then write your password in it, and protect it with:

chmod 400 secret

Then you can use this command instead:

curl --url 'imaps://imapserver.somewhere.net' --user 'username:<code>cat secret</code>'

If there is a SSL certificate issue, you can also try (not recommended of course):

curl --insecure --url "imaps://imapserver.somewhere.net" --user "username:password"

You need a newer version of curl, because version before 7.29 have limited imap support.

In order to upgrade curl on a Centos system, you can:

Create a new file /etc/yum.repos.d/city-fan.repo

Then paste the following contents:

[CityFan]
name=City Fan Repo
baseurl=http://www.city-fan.org/ftp/contrib/yum-repo/rhel$releasever/$basearch/
enabled=1
gpgcheck=0

Finally

yum update curl

You can very finally the version of curl:

curl --version

See also:

https://debian-administration.org/article/726/Performing_IMAP_queries_via_curl

Get url’s from website with wget

wget --spider --no-check-certificate --force-html -r -l2 https://website.of.interest.com 2>&1 | grep '^--' | awk '{ print $3 }' | grep -v '\.\(css\|js\|png\|gif\|jpg\)$' >> url-list.txt

which will give something like:
https://website.of.interest.com/en/Category:BRA
https://website.of.interest.com/en/Category:Free_software
https://website.of.interest.com/en/Category:Freeware
https://website.of.interest.com/en/Special:RecentChangesLinked/Software_overview
https://website.of.interest.com/en/Special:WhatLinksHere/Software_overview
https://website.of.interest.com/en/Talk:Software_overview
https://website.of.interest.com/ithelp/index.php?title=Software_overview&action=edit
https://website.of.interest.com/ithelp/index.php?title=Software_overview&action=history
https://website.of.interest.com/ithelp/index.php?title=Software_overview&action=info
https://website.of.interest.com/ithelp/index.php?title=Software_overview&oldid=2586
https://website.of.interest.com/ithelp/index.php?title=Special:Pdfprint&page=Software_overview
https://website.of.interest.com/ithelp/index.php?title=Special:UserLogin&returnto=Software+overview
https://website.of.interest.com/ithelp/index.php?title=Talk:Software_overview&action=edit&redlink=1

Match email address

If you have a file with several lines that contains email addresses that you like to extract, you can do that with Linux grep command:

File “emails.txt”:

somestreng anotherstring 342345 somename.lastname@domain.com

where somename.lastname@domain.com is the string you like to extract:

you can do this with:

grep -EiEio '\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}\b' emails.txt

Result:

somename.lastname@domain.com