Last Wednesday was my last day at work before a long weekend with my kids, who were home on spring break. At the end of the day I was scrambling to print out a product specification sheet to sign off on and return to the vendor before I left. I printed the document from MS Word to my Novell Netware server running iPrint and heard beeeeep… The signal of death from a Netware box.

I looked over and the cluster node it was on had abended in Apache2 with the LibC module being the culprit. These kinds of flaky things happen occasionally, so Power cycled the box (it hard locked, not letting me even write an abend log before restart) and waited for iPrint to roll ove to another node in the cluster. When it did, I printed again. Same result…. beeeep… Another cluster node bites the dust. Now I was pissed. This obviously wasn’t some corrupt PostScript font or something. I took the Word document and made it into a PDF, thinking this would clean up any funky code or fonts in the doc, then I tried to print it back to the original server that had now come back up and got iPrint running again… beeeep… ARGH!!

I lost patience and called the vendor to fax me the sheet so I could leave and took off for the long weekend convinced it was still some sort of document problem. Well today I had to print something for the first time this week and guess what… Crash and hard lock again on a different document. Now it was a apparent that it was a bigger issue so off to the Novell support website I run off to while servers reset themselves. What do I find but two TIDs on this issue.

• NetWare 6.5 server ABENDs in LIBC and Apache2 on iPrint server
• Unable to install iPrint printer from Macintosh workstations

The first TID explains that the recent Apple Security Update 2008-002 introduced problems with printing to iPrint servers, namely Abending and crashing the server. It offers patches to fix the Abend problem. The second TID explains that while the Abend problem can be alleviated, once applied the patches break the iPrint client’s ability to add printers to the computer from the browser and they need to be manually added through the Printer preference pane. This defeats most of the purpose of iPrint.

I’ll try to call Apple when I have a minute to see what they have to say about it.

To top it off, while trying to get this issue tracked down, I discovered that My tape jukebox is reporting an error… another problem for another day.

I ran across this news.com article on an experience the author had in submitting a bug report on the OS in the XO laptop his daughter was using to write a story. It is a prime example of how open source can be ultra nimble in responding to users while keeping strategic goals and distro stability in mind. I wish Apple would respond this way.

I have had Novell write custom patches for me before, but the process was nowhere as painless as this.

Blogged with Flock

Tags: OS, XO, BugReports, Apple, Novell

I’ve been monitoring the Apple related features in OES2 very closely, particularly the inclusion of AFP or an OS X Native NCP client with OES2. Today, Novell’s podcast called “Open Audio” published a show with a segment with Jason Williams, the project manager for OES2 where he talks about the inclusion of AFP. Here is my summary:

1. He and other Novell employees working on OES2 were absolutely shocked with how bad they got beat up by the beta testers and Brainshare attendees over the lack of AFP support in OES2.

2. They heard the multitude of voices and took it back to their upper management.

3. They have decided to include AFP support in OES2, but not until SP1 AT THE EARLIEST!!! GAH!!!

4. They are going to base their version of AFP on Linux on their AFP module on Netware.

5. They want to include additional functionality like the inclusion of resource forks.

6. They want your input on the features you need, and the versions of OS that you need to have supported.

Technorati Tags: OES2, AFP, Novell, Linux, SUSE, Jason Williams, Open Audio

Powered by ScribeFire.

Original Environment

• 1 x 6.5 iscsi targer
• 2 nw 6.5 cluster nodes
• 2 oes linux added to edirectory and to iSCSI storage

Finish with

• 1 6.5 iscsi target
• 2 OES linux cluster nodes

Netware admin

• installing adding nodes from nw dwployment manager
• cluster ocmmands on console
• management from console one, imanager and norm
• NSS Shared file systems
• Shared disks SCSI, SAN, ISCSI
• ISCSI initiator and target

On Linux

• Install add nodes from YaST
• Cluster commands in shell
• Management from iManager
• NSS and Linux native file systems
• Shared disks can be SCSi, SAN, iSCSI
• iSCSI target only.

Which filesystem is best?  If you are migrating from NW to Linux OES, NSS is the only choice supported on both platforms.  If the special features of NSS are not needed, it is relatively slow when writing new files.

• In migrations of services from NetWare to Linux, keep NSS
• Novell iFolder, iPrint , many AMP applications, GroupWise and Instant Messaging don’t need NSS
• Use ext3 if a non-cluster aware file system is needed
• Use PolyServe or OCFS2 (SUSE Linux Enterprise Server 10) if a cluster aware file system is needed
• Don’t use ext3 for GroupWise; it will get very slow because of the many small files that need to be written.  Instead user ReiserFS.
• ext3 will have indexing in OES2, then it will be OK
• The problem with Reiser is that it is very difficult to upgrade because each new version is almost a total rewrite.

Migration Scenarios

1. In Place Upgrade
2. Across the Wire Migration
3. Mixing Netware and Linux in one cluster to keep the current configuration (Recommended Solution)
1. With New Nodes
2. rolling cluster upgrade
4. Manual migrations / new installations

It is also possible to Install a new cluser and use Novell Server Consolidation and Migration Toolkit to copy data and file system rights.  Install a complete new environment and copy data, rights and apps with your own tools.

USING A MIXED CLUSTER IS NOT A PERMANENT SOLUTION!  
    It is only useful for migrations

NSS Storage resources can be loaded on Netware or NSS
    After adding a LInux node to NW cluster, it is not longer possible to add new NetWare nodes.  Only Linux nodes can be added.  In a mixed environment you can not add storage to the cluster or modify the existing storage pools without a total cluster reboot.

Mixed cluster limitations

• File system trustee assignments are calculated for NSS access via NCP the first time when a volume is migrated to Linux
• Trustees are stored in an XML file on the volume
• When rights are modified on the NetWare the Linux server will not automatically detect these changes and vice ersa.  (Update manually: ncpcon nss resync=VOLNAME)
• A volume can be successfully fail over from NetWare to Linux vice versa.   But when trustee assignments are added after the fail, they will not be recognized by Linux for NCP after  fail back.  So from a practical standpoint we recommend to leave the cluster volumes on Linux once you start the upgrade.

Not all applications can seamlessly migrate from NetWare to Linux.  Some need to be modified to run on Linux and must stay there after the conversion.  EX.   GW and iFOlder 2

• Cluster resource scripts will automatically be translated from NetWare commands to Linux commands
• If a resource is recated on a Linux node in a mixed cluster the resource will not run on a NetWare node.

Steps

• Add linux nodes to cluster
• migrate resources to linux nodes
• shows that it works to users
• remove nw nodes
• finalize cluster upgrade

1. On first linux node, go to yast, Novell Cluser Services
1. Join existing cluster
1. enter cluster FDN
2. Pick adapter that will have the heartbeat
3. Start cluster services
4. do a cluster view and it will show up at the command line
5. issue cluster status and it will show resources on the nw nodes
2. On nw cluster node 1 do a cluster-leave
1. resources will fail over to other nodes in the cluster
2. watch cluster status will update every 2 seconds (linux)
3. cluster migrate data_server oeslin1  to move the resources to linux node (from linux box)
3. On nw cluster node 2 do a cluster leave
1. cluster status
1. all volumes move to oeslinux1 server
2. resource scripts are stored in /etc/opt/novell/ncs
1. Are created automatically on creation of the node
2. can be moved into edirectory with a script
1. execute the cluster convert command to update them into edirectory
1. cluser convert preview resource_name
2. cluster convert commit
3. Create Linux specific resource templates
4. Add 2nd Linux node to the cluster
5. Remove netware nodes
1. in iManager on oeslin1
1. Go to clusters ->cluster manager
1. browse to the cluster node in cluster manager
1. go to edirectory managerment, cluster container
1. remove OES NW1 server and OES_NW2 server
2. unload cluser services
3. remove cluster load command from autoexc.ncf
4. Remove the servers from the tree in nwconfig

powered by performancing firefox

Technorati Tags: Brainshare, OES, Cluster, Netware, Linux, iSCSI, iFolder, GroupWise, iManager

I’m sitting here in the Wednesday keynote waiting for it to start (I usually don’t attend the Wednesday keynote and I thought I’d update my last two sessions from yesterday.

In LCT102 Laura Chappel’s “I hate this network” class we spent an hour and a half with wireshark and looking at about 8 different packet traces and try8ing to determine problems in them.  Maximum packet size issues, BitTorrent traces, and some good fun were all included.  What Laura does is amazing.  The only disconcerting thing about her classes is that she expects that everyone wants to be or is able to spend the time to be as comfortable with packet level analysis as she is.  While I’d love for that to be the case, it’s simply unrealistic for me.

In BOF 182, the K-12 Birds of a Feather meetup David Brower, the Education Vertical Manager led a discussion without slides.  It was a more free-form discussion where attendees were encouraged to raise issues that they were concerned about.  Issues raised included:

1. Why should schools use Novell Storage Management?
2. What is Novell doing to lower support call costs (CNE Discounts are returning)
3. Running SLED in a thin client environment
4. The status of running education applications in SLED including open source alternatives and application ports as well as using Wine.
5. Something related to the above comment was the discussion of the Centre student management suite.  One of the members in the discussion had switched to Centre and had some custom modules written for state reporting in New Hampshire.  I thought this was interesting because we are also looking at Centre.
6. The one I was most interested in and vocal about was regarding the status of AFP in OES2.  I really went off at this point.  I don’t think my comments were well received.  I ranted about the various failings of Novell over the years with regard to Apple, culminating with this lack of a Directory enabled version of AFP on OES2.  David started in on a story of a recent meeting he had with a senior vice president of Novell in which, allegedly, he was asked what Novell needed to do in the Apple space.  If this is true, it may finally be an acknowledgement from Novell that Macs are important to their business.  The word I’m hearing from people who are in the OES2 beta is that AFP is the number one item that is being asked for on their conference calls and meetings.  I find this hard to believe, but I really trust one of my sources.  I wish there wasn’t so much uncertainty here.
7. The final item that came up in the meeting was that David is going to work on setting up a technical listserv for k-12 customers so that they can help support each other.  This will be a closed entry list with approval only by educational reps.

powered by performancing firefox

Modular design and new manager

• supports large cluster up to 16 nodes tested
• Powerful resource dependency model (SML) automitically replicated
• Resource co-location Positive or negative
• Resource operation ordering independent of placement
• resournce can be depend on arbirtrary node attributes for placement (can failover to any box with more than 1GB of ram for instance)
• priorities:  higher priority resources will be satisfied first
• All rules can be time dependent
• Year, month, day of week, hours, minutes, phase of the moon
• Native support for migrating a resource in XEN
• Limited GUI
• Support for complex resource types
• Groups: simple group of primitive resources to simplify management
• Clones: Resources active more than once (OCFS2, Cluster-aware applications, Xen guests)
• Multi-state: Resources in more than one state, like preplicated master/slave services
• Resource agents for
• SAP
• Oracle, MySQL, DB2, Postgres
• Xen
• IBM Web Application Server
• Filesystems, Webservers, IP addresses, FTP, drbd
• Split-site clusters with external quorum

Technorati Tags: Novell, Brainshare, Suse, SLES, Heartbeat

powered by performancing firefox

By Amin Y Marts – Applied technology strategist amarts@novell.com

• Generic Backup Environment
• One backup server with attached tape device
• Backup targets of NW, Suse, and Windows targets with direct attached storage
• Internet SCSI Defined
• iScsi is an end to end protocol for transporting storage I/O block data over an IP network
• Common key advantages
• Builds on stable and familiar standards
• high degree of interoperability
• scales to gigabit speeds
• price per port
• Suse Linux Enterprise advantages
• Graphical iSCSI management tools
• Support of iSCSI as both initiator and target
• ISCSI Deployent
• Host Bus Adapters
• 85% of the market uses software initiators
• 10GB iSCSI requires host bus adapter
• Encryption
• Segregation of storage and network traffic
• vLan
• Physical segregation
• Misc
• Expect 10% overhead on typical server hardware
• iSCSI vs. Fibre Channel
• Fibre Channel
• LUN Masking
• Zoning
• Host Identification via World Wide Web Name
• iSCSI
• Physical and virtual network separation
• IP address access restrictions
• CHAP authentication
• Tape Drive Technology: Terms
• Tape Drives
• Linear
• Helical
• Azimuth
• Trajectory of an angle measured in degress goign clockwise from a base point
• Servomechanism
• mechanism designed to manipulate Read/Write Heads into the appropriate positions
• Goal of all of these technologies is to make sure a steady stream of data is available at the tape head.
• Archive and Backup – What’s the Difference
• Backup
• Collection of non-contiguous data sored for the tactical purpose of supporting the physical infrastructure
• Archive
• Collection of searchable stored data whose restoration aligned with strategic business goals or stressors
• Think the new shadow volumes with OES2
• Benefits
• Regulatory Compliance
• Storage Management and costs
• Backup Application Performance
• Protection against
• Litigation (data capture and production)
• Government Audits
• Open Records Request
• Knowledge Management (data mining)
• Features
• Robust indexing capability
• Robust storage medium compatibility
• Secure archived-data
• Access Logs
• Minimal End-user manipulation and training
• Non-IT access and management
• Configurable to corporate retention policies
• Novell Storage Services
• Access control with Granular Rights (vs. rwx)
• File salvage from the client
• Data Shredding
• Novell Backup Environment Devined
• Backup Target
• SMS (Storage Management Services)
• SMDR (Storage Management Data Requester)
• TSA (Target Service Agent)
• API’s
• 3rd Party Backup application
• Partner Issues
• Why are some backup software vendors better suited to Suse LInux and OES
• Many don’t backup NSS metadata
• Q and A
• Zmanda for sles backup (not OES)
• Open Source
• OES compliant
• Syncsort
• Symantec
• CA
• Commvault

Overall I was dissapointed with the quality of this presentation.  I was hoping for a discussion of unique issues surrounding Linux backup vs. NW Including volume types, boot volumes, and NSS and iFolder issues.  None of this was covered.

Technorati Tags: OES, OES2, Linux, Suse, Backup, Restore

powered by performancing firefox

• Oops
• Types of Issues
• System doesnt install
• Installation Workflow
• Syslinux or pxeboot loads the installation kernel initrd and starts the kernel
• The kernel mounts the initrd and starts linuxrc as the init process
• linuxrc sets up devices required to load the root system and loads it to a RAM disk
• The root system contains YaST, which will request additional information, prepare disk volumes and install packages
• linuxrc starts YaST and detaches the initrd with chroot_pivot
• After installation of the packages YasT starts the installed system with or without a reboot
• On the first start of the installed system yast gets called again and finishes lloading the system
• Installation Doesn’t work
• Check DVD, drive, or BIOS settings
• Bootload comes up, but cannont load kernel or initrd
• Check installation media
• Kernel starts bu hangs at splash screen
• Press F2 and check messages
• Still hangs at splass screen
• Reboot, start with textmode and safe mode options and check messages where it gets stuck
• linuxrcstarts, but hangs
• try safe mode boot options
• linuxrc starts, but falls back to manual setup
• Normal for mainframe.  Go through manual setup
• Cannot find installation system.  Check installation source.
• YaST2 starts, but eventuall crashes and falls back to linuxrc
• check if disks have correct partition label
• check if filesystems have been created
• Check if enough RAM
• Check installation source
• YaST2 still crashes
• Check /var/log/YaST2/y2log
• There is a shell on text console 2
• On zSeries there is an sshd process running to login via network during installation
• After Installation
• runlevel says unknown runlevel
• vnc/ssh/remote X11 installation
• YaST2 hasn’t run after installation
• vnc installation: start vncviewer
• X11 installation: start remote X11 server
• After Boot
• Kernel Boot Messages
• Linux kernel prints alot of status messages during startup
• While the amount might be confusing at firste, they are a valuable resource
• Some messages smell like error messges, but are in fact just harmless status notices or warnings
• the kernel sometimes finds flaws of the hardware design or BIOS and complains about them
• Kernel messages are logged by the klogd daemon, which feeds them to syslogd
• Previous kernel messages can be red with dmesg, whidch reads the /proc/kmsg ring buffer
• Boot Paramaters if the kernel starts but eventually locks up
• Failsave kernel paramaters
• ide=nodma – use PIO modes for ide disks – slow
• edd=off BIOS extended disk driver services off
• apm=off don’t use APM
• acpi=off dont use ACPI
• noapic  use XT interrupt controller
• nosmp  turn off smp support
• maxcups=0 max cpu usage is really 1
• Beware of the difference between ACPI and APIC
• ACPI is Advanced Configuration and Power Interface (provided by the bios)
• APIc is the Advanced Programmable Interrupt Controller, a part of the mainboard’s chipset
• To add to the confusion, the ACPI table contains the APIC configuration data
• ACPI=off turns off ACPI support
• system does install but doesn’t come up
• System locks up after a while
• System actually locks up or crashes
• Apps do not start
• Apps crash
• Expected better performance
• Everything works fine, but what does this message mean?

Ended up leaving the session due to irrelevant content to me… also very difficult to understand speaker.

powered by performancing firefox

Technorati Tags: SLES, Troubleshooting, YaST, YaST2

1.  New boss, Same as the old boss.  Unfortunately, the replacement of Jack Messman as CEO of Novell did not lead to any significantly better Keynote presentation.  B O R I N G!  Once you’ve seen a Stevenote, the rest is pretty pointless.

2.  OES2 is going to have some really amazing stuff in it, primarily in regards to XEN paravirtualization.  Clustering GroupWise or other apps through XEN should allow seamless cluster failover.. and when I say seamless I mean they are claiming zero packet loss when moving VM’s between two different physical machines.

3.  Bonsai is looking good.  The next version of GroupWise finally has a Mac/Linux client that looks usable.  I do need to talk with the GW folks at meet the experts about integrating with OS X address book (or at least importing it) since I have so many users already using that facility.

4.  iFolder 3.6 is looking good.  Unfortunately the Mac client will lag behind the release about 90 days.  I’ll have to keep my eye on that.

5.  AFP on OES2 doesn’t look very good.  Spoke with some guys at the NSS booth who handle filesystems and they said that they were shocked during the beta conference calls how many people were asking for AFP services.  I sent an email to one of the product managers involved in making that decision.  That email is below.  I encourage you to write the same email if you are interested in this feature.

    From:       ecallis@district106.net
    Subject:     AFP on OES2
    Date:     March 19, 2007 3:34:16 PM MDT
    To:       kkimball@novell.com

I’ll try to post some more notes today… BS 2007 Day 2

Technorati Tags: Brainshare, 2007, AFP, OES2, iFolder, SMB, Bonsai, GroupWise, XEN, paravirtualization

powered by performancing firefox

Just wanted to shoot a plug for a friend I met at Brainshare last year, Bert Platt.  His blog is over at http://www.bertplat.nl  He’s got some good BS posts and I’m sure he will all week long.

Technorati Tags: Brainshare, Blog, Netherlands

powered by performancing firefox