Seegras Logbook

Initially, I started with the premise to change my Console/Terminals to UTF-8. However, as it turned out, they behave rather weird, some things don’t work right, characters are sometimes not visible, they tend to make a mess in displaying special chars (as used in mc), some control-commands don’t work anymore and so on. My light-weight-terminal aterm does not support UTF8 (but JIS and whatnot, what the heck?) So I postponed this to some later date, when there’s at least a fixed mlterm available.

However, there are some interesting things to mention, in regard to the console:

mc clogs up the history
If Midnight Commander writes funny things like cd "`printf "%b" '\0057'`" into the shell-history, then HISTIGNORE is not set, or does not contain a space.

export HISTIGNORE="&:[ ]*:exit"

Umlauts appear as ? when doing an ls
This is due to the really relevant variable regarding the charset: LC_CTYPE. Set it to something other than “C”:

export LC_CTYPE=en_GB.ISO-8859-1

English system in Switzerland
If you want the system in english (and not necessarily the data), but are otherwise not english speaking, and sure as hell do not want screwed-up data formats or medieval measurement units, you want to set up your locales like this:

export CHARSET=ISO-8859-1
export LANG=en_GB.ISO-8859-1
export LC_CTYPE=en_GB.ISO-8859-1
export LC_MESSAGE=en_GB.ISO-8859-1
export LC_MESSAGES=en_GB.ISO-8859-1
export LC_NUMERIC=de_CH.ISO-8859-1
export LC_TIME=de_CH.ISO-8859-1
export LC_COLLATE=de_CH.ISO-8859-1
export LC_MONETARY=de_CH.ISO-8859-1
export LC_NAME=de_CH.ISO-8859-1
export LC_ADDRESS=de_CH.ISO-8859-1
export LC_TELEPHONE=de_CH.ISO-8859-1
export LC_MEASUREMENT=de_CH.ISO-8859-1
export LC_IDENTIFICATION=de_CH.ISO-8859-1
export LANGUAGE=en_GB

Setting LC_PAPER can lead to problems with things you do not send to a printer (pdf-generation for instance), and setting LC_ALL makes it impossible to use different locales for different uses.

Of course, you can replace ISO-8859-15 with UTF-8 if you like.

make mlterm look decent
These are the contents of .mlterm/main:

only_use_unicode_font=yes
scrollbar_mode = right
scrollbar_view_name = next
fontsize = 16
ENCODING = UTF-8
fg_color = white
bg_color = black
use_transbg = true
brightness = 60
fade_ratio = 75
geometry = 80x25
use_anti_alias = true
logsize = 4096
word_separators = " ,;=:"

Well, yes, it now looks quite NeXTy ;).

A better cursor
Not really console-related, but anyway: Get an Icon-theme like Shere Khan X (why, it looks like the one from NeXT of course), and put the unpacked directory into ~/.icons/ Now change your .Xresources to read:

Xcursor.theme: Shere_Khan_X

Screenlock with DPMS
For some machines, xlock is seemingly unable to do DPMS. Don’t fret:

xset dpms force off && xlock -mode blank -resetsaver

See Also
An earlier post in Blog: Useful Shell Oneliners

Microsoft Windows is a basically an 80ies operating system, aimed at non-networked single-user workstations with everything else just thrown in. And it shows. Add a lot of incredibly stupid “backwards compatibility” hacks and you get a nightmare.

• Kernel. Might be quite good, but it has one big deficiency: It’s not Unix. I doubt they will or can change that, tough it would help a lot in porting software.
• Filesystem: Case-insensitive but case-preserving is an utter fuckup and a security-nightmare too. You already shot yourself into the foot with IIS and this
• Fragmenting Filesystem: No Unix-Filesystem does fragment, so why does NTFS? (And don’t even get me started with (V)FAT; that one should be shot).
• Charset: Get rid of that fucking cp125x-charsets. Now! Everywhere! Make it impossible to choose it anywhere. Use UTF-8. And bid those “smart-quotes” and other non-standard crap goodbye.
• CR/LF. Do that CR away, this is not a typewriter anymore, I hope..
• Shell: Backslashes and Drive-Letters are a bloody nuisance to every (C-)programmer. Who in his right mind would choose the escape-character as a directory delimiter?
• Terminal: No, a Terminal should not be limited to 25×80. You need to be able to change the resolution, and you also need to be able to switch the charset. And copy & paste must work too.
• Control Characters: Misusing BREAK (Ctrl-C) as “copy” is a bad idea. Terminal control characters hsould never be used for anything else but their original use. You’ve got a Hyper- (Windows-) Key, use it.
• Mouse-handling: I personally can’t stand click-to focus. Not only I use sloppy-focuse, but I also want autoraise. Windows can’t do that, and it even collides with its dozens of modal dialogs. Oh, and on Unix you can do copy & paste with the mouse alone, no keyboard required…
• Modal Dialogs: Tons of them, most of them completely uselessly so.
• Registry. Please explain the benefit of this monster over config-files with a clearly specified structure.
• Incoherent separation of user-config and system-config (resulting from the registry). I should be able to take my personal config from one account to the other by act of simple copying.
• Missing desktop-features: multiple desktops of course.
• Look and Feel: Where is the problem of letting the user specify how his widgets should look? Especially if I don’t like this Fisher-Price-look. As far as I can tell, these are easy customisable. And yes, ALL of them should change their look simultaneously. No Funny business with a different look for Media Player, neither
• Localization. This is very bad. I should be able to change the language of the GUI on a click. But at least per user. And I should not have to download a different version of some patch or some service pack depending on the language I’m using.
• Over-Localization. The different versions for different languages suffer from another problem: Everything is localized that should not be localized, including macro-languages (!), “AW” instead or “Re” in e-mails (opposing RFC 2822, no less) and even system users (ever seen a user “wurzel” belonging to the group “rad” instead of root:wheel on unix? Well you get indeed “Administrateurs” on windows).
• Decent Editor. Per default. One where you can choose that the input uses CR, CR/LF or LF and saves only with LF. One where you can select a charset for input (and save as utf8 only), one which can open files up to at least 2GB.
• Directory-Structure. Only a fucking idiot would name the programs-folder “Program Files” — with a space in it, and what’s more, different in every language! Why not just “programs”? And more: Why is there such a mess in the windows-folder? and the windows/system folder? And why are users preferences and files there too??
• Missing Home. No clearly defined directory where the users data should go. Well, there is one now, but the applications don’t adhere to it.
• ACLs. An actually nice feature of windows — if the default ACLs weren’t so braindead. Who got the idea that users need to have write access to the root or the windows-directory??
• DRM. Either this goes out of Windows, or Windows goes out of the window. Digital Restrictions Management is not a service to the user.
• Registration. And especially the failing of it if your hardware changes.
• Standards: They exist for you to use them, not to invent stupid competing formats. Away with that WMA, WMV, DOC, XLS-trash. You can still support them, but store your information in open and standardized formats in the first place, like mpeg, mp3, ODF. I want to hear “You might loose some information if you store this Open Document Text in Microsoft .DOC-Format. Do you really want to do this?”
• OpenGL: Speaking of standards, it should use OpenGL like all other operating systems, instead of DirectX.
• TWAIN: An utter fuckup of a standard for scanners. Should have been ditched in favor of SANE a long time ago.
• Autostart: The one for changeable media. A security problem, of course. Ditch it. Not necessary, just to save one click. Yes, you can turn it off, but actually it should be impossible to turn it on at all.
• Mandatory Locking: A nightmare for doing backups or working with shared ressources.
• Package-System: There is none. Every application decides itself where it wants to install things and whether to tell the operating system what it has done. Not only a security- but also a maintenance-nightmare.
• Error Reports. They’re here to tell you what went wrong, not to obfuscate the fact that you did not think that an error might happen in the first place.
• Help. What about a help-system which is useful? What about documenting the programs (and their hitherto unknown commandline-switches and registry-settings)?
• Reboots. There is one reason, and one reason only, to do a reboot, and that is a kernel-upgrade. And no program-, library or driver-installation warrants one.
• Internet Explorer. Either you do it right (XHTML, CSS, DOM, EcmaScript), or throw that garbage out of the system.
• Outlook. Either you do it right (raw-text, charsets, proper quoting, pop3-handling), or throw that garbage out of the system.
• Active-X. Throw away without replacement. It’s an unfixable security nightmare.
• 32bits. Yes, it’s about time for the next version only to offer a 64bit-version.
• Monitoring. There’s a standard, it’s called SNMP. And you’re supposed to export your status there, especially if the application in question is Microsofts own directory- or mailserver. At the least, your SNMP-server should be able to be extended with the output of monitoring programs. The lack of things like these are precisely the reason Windows has no right to exist in an enterprise environment.
• DNS poisoning: The underscore “_” is not allowed in domain names. How come Active Directory still does this in 2012, and you can’t turn it off? Obviously, Microsofts DNS-servers do not belong onto the internet.

Every so often you happen to get some document, mostly one written with a certain software package from a company in Redmond, that looks pretty good but needs some minor property changed like the font for the default paragraphs. No problem, you open the “Stylist” in your OpenOffice and change the font for “Default”. Does not work. Hmm, the document indicates every paragraph uses the “Default” preferences. And then, you realise that every paragraph has individually a font set.

Impossible to clean up that mess in the office-software. You can’t click “Clear formatting” for every paragraph, and besides, you would screw up any other formatting like bold faces and italics too. XML to the rescue! In theory it should be possible to unzip the document and edit the XML. As it happens, the several megabytes big XML is very structured indeed: Everything is on one line Thank you very much, this means most normal text-oriented unix-tools won’t work, because you can’t rely on some useful delimiter.

Luckily I found xmlindent which nearly does the job, you can get nearly the original XML (with the exception of one missing linefeed after the XML-declaration) with sed s/\ \ \ \ //g and tr -d "\n" afterward you’ve done editing. Also interesting is Editix a Java-based XML-Editor.

Now, I would like to get rid of 10’000 redundant style-definitions, which either define bold or italic or are used to set small caps and bold to designate a subtitle — preferably set those who define title to a real “Heading”. sed -n '/<style :style style:name=\"P/,/<\/style:style>/p'</style> will give me the whole statements, but what now?

I needed some new hardware to keep my appointments sorted, my address-database, and things like that, and I wanted those things encrypted. Instead of opting for some smartphone or PDA-type hardware, I decided on the Asus EeePC subnotebook, which costs about the same, or even less than modern smartphones or PDAs.

I got mine from digitec, a german edition, since it wasn’t available in switzerland yet. After playing a bit with the installed Xandros I decided to install Debian, since I found I was lacking packages, and I wanted to encrypt /home anyway. I decided not to change the partition-tables, and to put Debian on /dev/sda1 solely. In xandros, the system was on /dev/sda1, mounted read-only, and /dev/sda2 was a union-mount onto it. In hindsight, this wasn’t a bad choice, I really needed the 2.5GB to compile kernels..

I installed pretty much according to the DebianEeePC Howto and then started compiling my own kernels. You need the AR2425-patch, and the 2.6.24-patch from here: http://madwifi.org/ticket/1679 in order to get the wireless working. Then you’ll probably also want a driver for the ATL2-ethernet-NIC, version 2.0.4 works with 2.6.24.X-kernels. Sadly, the ATL2 only works when loaded as module. And for the special buttons to work, you’ll want to apt-get install eeepc_acpi. Well anyway, what you’ll want is my .config for Asus EeePC.

The xandros on the eeepc boots tremendously fast, most of which can be traced back to their fastinit-initd, which has been reverse-engineered, by the way: fastinit reimplementation. A short look at this and “strings /sbin/fastinit” reveals that it only does the minimun, and starts X as user “user” with just a login shell. Sadly, this does not work if you’re going to encrypt /home with dm-crypt. After looking at some alternatives, I settled on KDM with a pleasant looking-theme on my own, based on Kurumin KDM1 (without girl and swirl).

The login uses pam-mount to automatically do cryptsetup luksOpen for /home. /etc/security/pam_mount.conf.xml needed to get this line added:
<volume fstype="crypt" path="/dev/sda2" mountpoint="/home" />
and /etc/pam.d/kdm and /etc/pam.d/login each got the line
@include common-pammount
attached at the end.

I had made a backup-copy of the whole flash with dd and nc over the network, so I could just re-use some configfiles such as xorg-conf.

Other noteable specialities are some defaults which make more sense for flash-based systems, like mounting filesystems with noatime. This is my fstab:
proc /proc proc defaults 0 0
/dev/sda1 / ext3 noatime,nodiratime,user_xattr,errors=remount-ro 0 1
tmpfs /var/log tmpfs defaults 0 0
tmpfs /tmp tmpfs defaults 0 0
tmpfs /var/tmp tmpfs defaults 0 0

As for the window manager, I tried out several environments. icewm (was used on xandros), fluxbox, openbox, matchbox and finally xfce4. I’m still not quite satisfied with it.

From time to time you come upon some useful oneliner in the shell. Either because you see it somewhere, see someone doing it, or because you need it and produce one yourself. And most of the time, it’s not enough to put into a shell-script, so you find yourself hunting for it in your .bash_history. Well, here are some:

Do something to a lot of files
for i in *; do command $i; done

You’ll need this a lot. Does not work with files containing spaces

Convert a load of images
for i in *.tif; do convert -quality 75 $i `basename $i .tif`.jpeg;

Make pdf out of a several images
convert -limit memory 32 -limit map 32 *.png target.pdf

You really want to set the limits, unless you have more RAM than the whole images converted to pbm.

With Graphicsmagick, you need to set a compression for jpg, otherwise the resulting PDF will be huge.

gm convert -compress JPEG *.jpg target.pdf

The resources you want to give those programs can be set using environment-variables:

export MAGICK_MEMORY_LIMIT="640mb"
export MAGICK_MAP_LIMIT="320mb"
export MAGICK_AREA_LIMIT="640mb"

Make pdf out of a lot of images
But the whole idea of using ImageMagick or GraphicsMagick to create pdf-files is very much flawed, since both first convert the jpeg-files into an raw, uncompressed intermediate format which tends to eat up all available memory, and then crash. And which is completely nonsensical too, since jpeg-files can be embedded into pdf as they are. Luckily, pdfjoin will help us, tough it accepts only files with the extension ‘.pdf’ as input as of yet:

mmv '*.jpg' '#1.pdf'; pdfjoin --outfile target.pdf *.pdf

Change resolution (DPI) of an image
convert -density 600 -units PixelsPerInch source.png target.png

split/crop pictures
convert -crop 1230x880 source.png target

Note that this does not really crop the image, but splits it into as many parts as necessary, using “target” as prefix, adding a number as suffix.

fix offset
convert +repage source.png target

Lately, “crop” seems to behave differently, it leaves all but the first part with an offset. This makes the picture disappear in gimp, and stretched if you make a pdf out of it. That’s why you may need to repage.

rotate
convert -rotate 90 source.png target

rotates 90 degrees clockwise.

Attach pictures to each other, one below the other
montage +frame +shadow +label -tile 1x2 -geometry "widthxheight+0+0" source1.png source2.png target.png

Important is that width and height are the dimensions of each of the sources. For putting them next to each other, just use 2×1 as tile.

Rename files according a textfile-list
for i in `cat list` ; do mv `echo $i | awk -F"-" '{print $1".ext"}'` $i.ext ; done

Delete empty directories
find . -type d -empty -depth -exec rmdir {} \;

Remove first page of a PDF
pdftk A=source.pdf cat A2-end output target.pdf

Yes, pdftks syntax is a bit alien.

Merge PDFs
pdftk sources*.pdf cat output target.pdf

Unpack lots of packed files into different directories
for i in *.rar; do mkdir `basename $i .rar`; mv $i `basename $i .rar`; cd `basename $i .rar`; rar x $i; cd ..; done

If you have a load of packed files, in this case rar, and you want to unpack each into a different subdirectory, according to the name of the rar-file.

Get rid of magnatune-advertisements in mp3s

for i in *.mp3; do mp3splt -s -p min=2 -d `basename $i .mp3` $i ; done

Merge AVI-movies
mencoder -forceidx -oac copy -ovc copy -o outputfile.avi parts*.avi

That’s the basic way. If you want to process a whole directory with avi-files whose filenames have a distinct part (like “cd≶num>” or “-Part≶num>” as in my example) you can do something this:

for i in *-Part1.avi; do mencoder -forceidx -ovc copy -oac copy -o `basename $i -Part1.avi`.avi `basename $i -Part1.avi`-Part*.avi; done

To be continued. You might also want to check out my Program-section for small useful scripts.

Since I spent so much time in finding out why the hell sshd kept crashing when I turned on ldap-support, other people might as well profit from it.

This is what happened after I turned on ldap-support in /etc/nsswitch.conf and /etc/pam.d. According to auth.log:

Jun 6 13:59:12 proto sshd[27433]: fatal: buffer_put_cstring: s == NULL

messages also knew something:

Jun 6 13:59:12 proto kernel: pid 27435 (sshd), uid 0: exited on signal 11
Jun 6 13:59:12 proto sshd[27433]: fatal: buffer_put_cstring: s == NULL

An sshd segmentation fault when trying to log in. According to what I’ve found in bug-reports, it does this if ldap.conf or nss_ldap.conf does not exist. Only these did exist in my case. debugging-output from nss_ldap turned up nothing, pam_ldap doesn’t even support a debug-flag: “This option is recognized by pam_ldap but is presently ignored.” Says so in the man-page. Thank you.

The key to it was revealed by putting in “debug 5” into ldap.conf and starting sshd with debugging:

# /usr/sbin/sshd -d -d -d
[snip]
ldap_ndelay_off: 7
TLS trace: SSL_connect:before/connect initialization
TLS trace: SSL_connect:SSLv2/v3 write client hello A
TLS trace: SSL_connect:SSLv3 read server hello A
debug1: do_cleanup
Segmentation fault: 11

Well, it seems you need either to use ldaps://servername and not ldaps://IP in ldap.conf so SSL knows which certificate needs to be used, AND it needs a correct certificate. On the other hand, if I just use ldap://servername without SSL, then local ssh works again, however I seem to have a problem with all the ldap-services.

Some debugging of an sshd core-file reveals the following:

#928 0x0806820f in sshpam_thread (ctxtp=0x8079f80)
at /usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/auth-pam.c:469
Previous frame inner to this frame (corrupt stack?)

If you’ve got irresponsible people on your server who let their mailform get abused by spammers, you might end up on blacklists. This is what happens if your server lands on a blacklist of hotmail.com. Stupid “smart” quotes, excessive spaces and gracious html in the mails from Microsoft have been fixed.

After being directed to http://postmaster.msn.com/ you can fill out some form going to abuse@hotmail.com. Like this:

So you send mail to abuse@hotmail.com:

Hello, hotmail.com is blocking mails from [IP]. We’ve deactivated formmail which was misused for sending spam. Please unblock [IP].

You get an automated reply:

Thank you for reporting spam to the MSN Hotmail Support Team. This is an auto-generated response to inform you that we have received your submission. Please note that you will not receive a reply if you respond directly to this message.

Unfortunately, in order to process your request, Hotmail Support needs a valid MSN/Hotmail hosted account.

We can help you best when you forward the spam/abusive mail as an attachment to us. The attachment should have full headers / message routing information displayed. This means that the complete “From” address of the offending message should be displayed. If you need help to do this, please visit the following website:

http://safety.msn.com/articles/junkmail.armx

If you have deleted or no longer have the message, you can still resubmit your report by sending the name of the violating MSN/Hotmail hosted account and a description of your concerns. If your submission does not involve a third party, please include your own account name in the body of your message along with the description of your concerns so we can process your report.

For further instructions on how to submit spam and abusive emails to Hotmail, please visit:

http://postmaster.msn.com/cgi-bin/dasp/postmaster.asp?ContextNav=Guidelines

For more information about MSN Hotmail’s efforts and technologies used to fight spam and abusive e-mails please visit:

http://postmaster.msn.com/cgi-bin/dasp/postmaster.asp?ContextNav=FightJunkEmail

Well, you go to http://postmaster.msn.com/ and fill out the form:

Service: MSN Hotmail
What type of problem do you have? I need something fixed [Other]

Full Name:Peter Keel

What e-mail address would you like a response sent to? abuse@mydomain

Primary e-mail address/member ID associated with the account you are inquiring about:

Be specific when describing your problem. The details that you include
enable us to promptly send you the most likely solution to your issue.

On the 14.1 a user on [IP] sent out spam, with which we dealt on the 15.1.

A week later, you’re still blocking [IP], and we and our other
users would be obliged if you could unblock the server.

Frequency of the issue: Always present

How do you access your Hotmail account? Other

Who is your ISP? Other

Type of Internet connection: Other

Have you recently installed any new software (if you enter yes please
add more comments in the text box above)? No

Suprise, It’s answered by a person:

Hello Peter,

Thank you for writing to MSN Hotmail Technical Support. My name is Jervin and you mentioned that a Hotmail user is being blocked from sending messages. I understand how important it is for you to have this concern attended immediately.

I have checked the IP address [IP] associated with this account and found it to be functioning properly. The user should be able to send any e-mail message to any address without problems.

Please try to send a test e-mail message and see if you will still encounter the same issue. If for some reason the issue persists, kindly reply and provide a copy of the bounce message you have received after sending the test email. Please also provide the MSN Hotmail account in question.

As one of our valued customers, your satisfaction is one of our primary concerns. Thank you for using MSN Hotmail.

Sincerely,
Jervin
MSN Hotmail Technical Support

He didn’t understand what I was talking about. Reply:

Hello

“MSN Hotmail Support” wrote:
> I have checked the IP address [IP] associated with this
> account and found it to be
> functioning properly. The user should be able to send
> any e-mail message to any address without problems.

… Its the other way round…

> Please try to send a test e-mail message and see if you will still
> encounter the same issue. If for some reason the issue persists, kindly
> reply and provide a copy of the bounce message you have received after
> sending the test email. Please also provide the MSN Hotmail account in
> question.

This is one of _hundreds_ of rejects from today:

2007-01-23 14:36:50 1H9LpS-0002xm-Hu < = user@server41.mydomain
U=user P=local S=978
2007-01-23 14:36:51 1H9LpS-0002xm-Hu ** some_user@hotmail.com R=lookuphost
T=remote_smtp: SMTP error from remote mail server after MAIL
FROM:user@server41.mydomain SIZE=2033: host mx3.hotmail.com
[65.54.244.72]: 550 Command rejected for policy reasons. For troubleshooting
information, go to http://postmaster.msn.com
2007-01-23 14:36:51 1H9LpT-0002xw-H7 < = <> R=1H9LpS-0002xm-Hu U=mailnull
P=local S=2076

And of course, it does not just concern that one user of yours, but dozens.

Kind regards

Peter Keel
System Administrator

And I get another answer:

Hello Peter,

Thank you for writing back to MSN Hotmail Technical Support. My name is Angelie and I have read your e-mail exchange with Jervin regarding your issue on not being able to send e-mail to Hotmail users because your message bounced back to you. I know how important it is to have your concern attended to immediately.

Peter, in order to help you with your issue, please visit our Postmaster site at http://postmaster.hotmail.com/Troubleshooting.aspx. This site provides troubleshooting information to those who are having issues sending e-mail messages to Hotmail. If you need further assistance with your e-mail delivery issues, you need to click on the second “here” link under “Sender services, tools, and issue submission” or submit the necessary information at: http://support.msn.com/eform.aspx?productKey=edfsmsbl&page=support_home_
options_form_byemail&ct=eformts

I appreciate your understanding and patience.

Sincerely,

Angelie
MSN Hotmail Technical Support

I fill out this new Form:

Service: Sender Information Form

What type of problem do you have?

Primary company contact name: Peter Keel

Primary company contact email address: abuse@mydomain

What domain are you sending from? various

What are the IP addresses of your outbound mail server (as seen by the receiving mail server)? [IP]

Is your server dedicated or shared? Dedicated

What ISP are you using? Ourselves, we’re an ISP

What OS are you using? FreeBSD

What mail transport software are you using? Exim

What mailing list management software are you using? irrelevant

How are bounce messages (non-delivery reports) handled by your system? delivered to the sender

How frequent are your mailings? does not apply

What are the volume of your mailings? does not apply

What are some of the accounts on Microsoft’s systems that you are attempting to send mail to? some_user@hotmail.com and probably hundreds more.

Do your SMTP logs show any failed transactions when attempting to send
to addresses at MSN, MSN Hotmail or other MSN Services? If so, include
those entries here.
2007-01-23 14:36:50 1H9LpS-0002xm-Hu *= user@server41.mydomain
U=user P=local S=978
2007-01-23 14:36:51 1H9LpS-0002xm-Hu ** some_user@hotmail.com
R=lookuphost
T=remote_smtp: SMTP error from remote mail server after MAIL
FROM:*user@server41.mydomain* SIZE=2033: host mx3.hotmail.com
[65.54.244.72]: 550 Command rejected for policy reasons. For
troubleshooting
information, go to http://postmaster.msn.com

Can you telnet to port25 of mx1.hotmail.com from your mail server(s)? yes

Can you traceroute to 216.32.183.201 from your mail server(s)? yes

Please copy/paste samples of a few of the messages you’re sending – including full headers – as seen by your recipients. forget it. I’m not going to sift those out of the queue

In what manner(s) are recipients added to your mailing list(s)? does not apply

Provide the URL of your web site. does not apply

Provide the URL of your Privacy Policy. does not apply

Provide the URL at which users opt-in to receive your mailings. does not apply

Provide the URL at which users may choose to permanently unsubscribe from your mailing list. does not apply

Are you currently a Return Path client? No

Are you Sender Score Certified? No

Do you publish Sender Policy Framework (SPF)/Sender ID records for your IP? Yes

Do you use separate IPs for your transactional vs. promotional/subscription marketing communications? No

Are you in the Junk Mail Reporting Partner Program? No

Do you use Smart Network Data Services? No

And I sent a Mail:

Hello

“MSN Hotmail Support” wrote:
> Peter, in order to help you with your issue, please visit our Postmaster
> site at http://postmaster.hotmail.com/Troubleshooting.aspx. This site
> provides troubleshooting information to those who are having issues
> sending e-mail messages to Hotmail. If you need further assistance with
> your e-mail delivery issues, you need to click on the second “here” link
> under “Sender services, tools, and issue submission” or submit the
> necessary information at:
> http://support.msn.com/eform.aspx?productKey=edfsmsbl&page=support_home_
> options_form_byemail&ct=eformts

Alright, I did.

Still, the problem is a simple as it can be: Hotmail deliberately blocks Mail sent by SMTP from [IP]. Would you just please unblock that IP?

Regards

Peter Keel
System Administrator

Now I’ve got two answers, one on the form, the other on the mail,

Hello Peter,

We have identified that messages from your IP (<ip>) are being filtered based on the recommendations of the SmartScreen filter. SmartScreen is the spam filtering technology developed and operated by Microsoft. SmartScreen is built around the technology of machine learning. SmartScreen’s filters are trained to recognize what is spam and what isn’t spam. In short, we filter incoming emails that look like spam. I am not able to go into any specific details about what these filters specifically entail, as this would render them useless.

However, we also base our spam rating on the reputation of the sender. One way to positively impact the reputation of your IP is to obtain SPF/Sender ID records. This technology allows SmartScreen to better track emails from your IP, weeding out spoofed messages. You can find additional information on creating SPF records at http://www.microsoft.com/senderid

Please confirm that your emails comply with MSN Hotmail’s technical standards. This information can be found at: http://postmaster.live.com/Guidelines.aspx http://advertising.msn.com/adproducts/Email_TechStd.asp

It will also be helpful to send example emails to randomtestacct@hotmail.com from your MTA so that we can examine the types of emails that you send. This will help me in our investigation. When you do this please send me the subject lines of emails you have sent. Please make sure that these emails are functionally similar to the types of emails that originate from your system, as any deviation from this may hinder our ability to investigate your problem. Do not use the word, “Test” or anything that would change the email from the original message that was initially sent to Hotmail Customers.

I hope that the information that I have provided to you has been helpful. You may also be able to find additional information on common delivery questions at the Hotmail Postmaster Site found at: http://postmaster.msn.com/. I would like to highlight some key areas which I believe are appropriate to your company.

* Hotmail has created the Smart Network Data Services program. This is a service that helps legitimate email senders work with their customers and partners to reduce spam originating from their IP. http://postmaster.msn.com/snds/ This program allows a sender to monitor the “health” of their IPs.

* I would also strongly recommend that you authenticate your outbound email via Sender ID and publish your SPF records for your outbound email IPs and register with Sender ID. You can find additional information at http://www.microsoft.com/senderID or you may email senderid@microsoft.com for support. Please note that technical standards (RFC 4408) discourage use of “ptr” for performance and reliability reasons.

* Monitor user complaints. Hotmail also has a sender complaint feedback loop program Junk Email Reporting Program (JMRP). Enrollment in this, free of charge, program will benefit you as a sender as it will keep your e-mail lists updated and populated with interested MSN Hotmail Customers. Participation in this program will remove those MSN Hotmail Customers who do not want to receive e-mails from your company. If you are interested in joining this program please email HMJMRX@microsoft.com

While using the SNDS tool, enrollment in the JMRP or having your IPs registered with Sender ID will not allow emails from your mail servers to bypass our filters, these are in place to help legitimate companies deliver their emails to Hotmail Customers.

* SenderScore Certified Mail Program. Many legitimate mailers and marketers have qualified and joined this “white listing” program to improve mail deliverability and decrease email from being filtered to the Junk E-mail Folder. Sender Score is a third party program,
administered by Return Path. Sender Score (www.senderscorecertified.com) is the only White Listing service to which we subscribe.

The troubleshooting steps in this email are recommendations only. Microsoft makes no guarantees that following these steps will guarantee deliverability to MSN, Hotmail, or Live.com customers.

Thank you,

Tyler
MSN Hotmail Domain Support

Dear Peter,

Thank you for writing back to MSN Hotmail Technical Support. This is Noel and I am writing in response to your request to unblock your IP [IP]. I understand how important it is to have this IP unblocked since most of your e-mail subscribers are not able to send e-mail
successfully to our accounts. I am sorry for the inconvenience this may have caused you.

Peter, I would really like to this for you since unblocking is such a simple task. However, I would like to guide you that our technical facility does not provide this option. Unblocking of IP or removing them from Blacklist and adding them to Whitelist, are done by the Domain
Delivery Support Team. In as much as I would like to remove the IP [IP], I am not able to do so since we do not have access to the Domain Delivery Support Team facility.

To correct this, kindly ask assistance from the Domain Delivery Support Team. Visit our Postmaster site at http://postmaster.hotmail.com/Troubleshooting.aspx. The e-mail address they will be sending the answers to, is also specified on that page. Please accept my apologies for not being able to assist you fully.

On behalf of our entire Hotmail staff, a heart-felt appreciation for your understanding.

Sincerely,

Noel
MSN Hotmail Technical Support

In other words, back where I started. And I’m not talking to employees of that company again. They’re all very polite, mostly clueless, apologetic and in any case can’t solve a problem which takes takes one line from me (“please unblock [IP]”) and one word from the admin (“Done”) of another big mailservice.

Addendum: Seems that this problem is more widespread than I thought: The Register: Hotmail Friendly Fire.

In fact, I can confirm this, I sent (nearly) identical mails from 60 servers to a hotmail test-address. The mails were sent with the envelope-from of an account and the server FQDN as domain, and the header-from of an other existing account on another server. This is what the outgoing-log says:

2007-07-18 14:35:58 1IB8l4-000Pti-IX < = testhost@server.domain U=root P=local S=449
2007-07-18 14:35:59 1IB8l4-000Pti-IX => testaccount@hotmail.com R=lookuphost T=remote_smtp H=mx1.hotmail.com [65.54.245.8]
2007-07-18 14:35:59 1IB8l4-000Pti-IX Completed

Some 15 of those 60 mails were classified as “Junk”. However, one of those mails never arrived, nevertheless the log clearly shows it was sent, and was accepted by hotmail. I sent another mail from the same server, which also did not arrive.

I finally decided to replace my Firewall, an UltraSparc 5 running Linux with something which is smaller, consumes less energy and produces less noise. I cam upon the PC Engines WRAP which is with 15x15x2cm considerably smaller than the Sparc. It has a serial port, three ethernet-ports and runs its OS from a Compact Flash II Card.

There’s a tutorial on Installing Debian on a USB flash with everything encrypted. After a first try, I decided to leave out the “encrypted” part, its complicated and I have no decent way of entering any passwords at boot.

For the WRAP, there some things which differ from some USB-media wich you have to take care of:

There is no real-time clock. Therefore you need to turn off the filesystem-check by time gone unchecked:

tune2fs -i 0 /dev/whatever


Clock-source should be pit or jiffies, with scx200_hrt I got me a running clock of a one-minute deviation per second.

echo pit jiffies > /sys/devices/system \
/clocksource/clocksource0/current_clocksource


You want to minimize writes on the Flash, thus you best put /var/lock, /var/tmp, /var/run and /tmp onto a tmpfs, as described in the above tutorial.

The Kernel sometimes is a bit too verbose, this will fix that:

echo 5 > /proc/sys/kernel/printk


It’s also slow, and the new apt feature to download diffs of the package-indices does heavily use disk and cpu to save bandwidth. Not a good idea in this case. Turn it off in /etc/apt/apt.conf:

Acquire
{
Pdiffs "false";
};


Finally, you might want a kernel which does only that which is needed, loads no modules (which puts the brake on some rootkits which want to install themselves as kernel-modules). You can try and err, but as it happens, somebody else (me) already did, so heres the .config for a pc engines WRAP, kernel 2.6.18 with the grsecurity kernel patch.

I took a further look at charsets in mail, in respect to what I need to receive, and what spammers use. In theory, a mailer should always use the least necessary charset, us-ascii that is, unless the user types some non-ascii sign, in which case it should use the ISO-8859 charset if appropriate, or UTF8. If you configured it to use, say 8859-1, and you type an umlaut, it sets the charset to ISO-88859-1, if you type a cyrillic character, it should use UTF8. If you configured ISO-8859-5, it should use this for cyrillic, and UTF8 if you type an umlaut. Simple. So you only need us-ascii, ISO-8859 and UTF8.

Now there are some braindead and/or obsolete mail-programs, which use different and outdated charsets. As it happens, you will have contact to people whose mails appear in a hodge-podge of non-standardized charsets, most notably windows-125X. Now, for a western-european german-speaking context, where most mails are either german or english, with some very little french or spanish thrown in, I did some statistics regarding the charsets of spam and ham.

This is a sample of 1220 legitimate Mails:

83.6% iso-8859-1
5.3% us-ascii
4.6% utf-8
4.5% iso-8859-15
1.5% windows-1252

The rest half percent is negligible and consists of some other iso-8859 charsets.

Now with spam, this looks quite different, the sample here are 6251 spam-mails:

15.5% iso-8859-1
39.5% us-ascii
00.6% utf-8
00.1% iso-8859-15
14.7% windows-1252

Now, where’s the rest? Its mostly a huge amount of the windows us-ascii replacement windows-1250, which is a completely superfluous charset:

23.9% windows-1250
1.5% iso-2022-jp
1.3% koi8-r
1.1% iso-8859-2
0.5% windows-1255
0.3% windows-1254

The remaining percent are japanese, chinese and russian charsets, plus the remaining windows-125X-charsets.

So now I can get rid of 25 percent of spam by not allowing chinese and other east-asian charsets, russian koi8 and most windows-125X charsets except windows-1252. I could get rid of another 14.7% by blocking this also, but that would piss of 1.5% of my legitimate contacts.

The blocking can be accomplished by simply putting some rule into .procmailrc:

:0:
* ^Content-Type.*windows-1250.*
spam


Or by giving it a score in spamassassin

//begin
header __ILLEGAL_CHARSET_1 Content-Type =~ windows-1250/i
meta ILLEGAL_CHARSETS (__ILLEGAL_CHARSET_1 + __ILLEGAL_CHARSET_2>= 1)
score ILLEGAL_CHARSETS 3
describe ILLEGAL_CHARSETS foreign obsolete charsets


Or by refusing it completely during transmission in postfix’ /etc/postfix/headercheck.pcre

/^Content-Type:.*\bcharset="?(?:
windows-1250 |
windows-1251 |
windows-1253 |
windows-1254 |
windows-1255 |
windows-1256 |
windows-1257 |
windows-1258 |
windows-874)\b/ REJECT Illegal Charset


/etc/postfix/headercheck.pcre can also be used to block very selectively; if for instance an “unsubscribe” does not get honoured (it’s of course useless against normal spammers since they constantly change addresses):

/^From: .*test@example.com/ REJECT I said NO, you spamming moron


And if you want to test a rule, save an email as “testcase” or something and check if it matches:

postmap -q - pcre:/etc/postfix/headercheck.pcre < testcase

According to observations by Dave Piscitello and Bruce Schneier, that the traditional 4-fold model (authentication, authorization, availability, authenticity) is becoming inadequate, and as proposed by Dave Piscitello here is the new one:

Admissibility (what may connect)
Authentication (who are you)
Authorization (what are you allowed to do)
Availability (is the data accessible)
Authenticity (is the data intact)

I consider the adding of admissibility an extremly important change, given the large numbers of people who increasingly are not anymore in control of their own machines due to viruses, worms and digital restriction management (DRM). This is why I repeat this here.