Podcast
×
Free Online Registration Required

The tutorial session you want to view requires your registering with us.

It’s fast and easy, and totally FREE.

And best of all, once you are registered, you’ll also have access to all the other 100’s of FREE Video Tutorials we offer!

 

×
Transcript

Introduction & Agenda

Introductions & Agenda for ZFS File System Tutorial (Part 1 of 6)

 

>> Mick:  As Dave said, the webinar is looking at the software that is used to manage the Oracle 7000 series Storage Appliances which are based on the ZFS File System, which I’ll give you a little bit of an overview very shortly.

 

[pause]

 

The subtitle for this webinar is “How Easy is This? What More Could You Need?” As hopefully I will demonstrate to you how friendly the management interface is and the meeting in other facilities that are available to you and the software is the same right across the entire ZFS Storage Appliance range. What you’re going to see here is relevant to any of the systems that you might come across.

 

It’s a range of systems and Storage Appliances that are constantly being updated. In fact, there’s a new software update that is just coming out fairly recently with lots of new features. Because it’s ZFS-based, we’ll be looking, first of all, in a little bit about the backgrounds of the file system. A quick look at some of the models of the Storage Appliance that are available, how they are connected, the various capacities they support, the file sharing protocols, management protocols, etc. and some of the features that are unique to the 7000 series when compared to competitor systems. The main part of the webinar is a practical demonstration of the management software.

 

Right from taking it out from the box and switching it on up to the creating shares, users, and carrying out analytics to see how the performance of this system is working. One of the key features of the system is that it uses advanced techniques to provide very clear and in-depth analytics to monitor the workload that the system is undergoing and to enable you to hone in on any bottlenecks that you might find and any other problems, etc.

 

[pause]

 

Please ask questions via your chat window anytime. There will be certain breaks during the webinar when things take a little bit at the time to complete, which we can use as an opportunity to answer your questions. If not, even if we can’t answer things on the spot then we’ll get back at you, and rest assured that we’ll answer your questions one way or the other.

 

Copyright SkillBuilders.com 2017

×
Transcript

Introduction to ZFS What is ZFS

Introduction to ZFS – What is ZFS (Part 2 of 6)

 

>> Mick:  ZFS, as I say here, is the file system in which this Storage Appliance is based. As you could probably guess, the Storage Appliance is based on the Oracle Solaris operating system with a nice graphical and command line interface to manage it, which you’ll see very shortly.

 

ZFS is Zettabyte File System that was originated by Sun before it became part of Oracle and introduced into Solaris 10 in 2006. In fact, it started live 10 years ago in terms of the development was taking place.

 

It’s named Zettabyte because of its capacity which is pretty well unlimited. You could connect all the disks in the world together and it wouldn’t come anywhere near the capacity that would be theoretically possible for one individual ZFS file system.

 

[pause]

 

The capacity of the storage device is actually 256 quadrillion zettabytes, whatever that might be. Somebody might be able to work that out. But to give you an idea, the entire production of Seagate disks during this year is about 1/3 of the zettabyte.

 

Basically, all the old limitations that you have with other file systems and you can’t have any other file this big and you cannot have these many directories, etc. They’ve all been wiped away by ZFS.

 

[pause]

 

ZFS is still very much under development and they’re constantly coming out with new facilities for it. Of course, it’s available within Solaris 10 itself and Solaris 11 system which was released back in November.

 

[pause]

 

There are a few key features that this Storage Appliance has compared to other systems that make it standout a little bit. It has something called Hybrid storage pools, which is transparency. But it’s depending on how the system is configured. You have DRAM, flash storage (SSDs) and disks which are usually high-performance or high-capacity. Very fast access times for both read and write.

 

You have all the ZFS facilities also so rapid snapshots, clones – a vast variety of [3:04 inaudible] type facilities like triple parity RAID if you want to, triple mirroring, replications, thin provisioning. We can create a LUN that doesn’t actually occupy any space until you start putting data into it, for example. The nature is very secure because of techniques like data checksumming that ZFS employs. There’s also built-in compression of various levels, de-duplication, etc.

 

Last but not least, dtrace analytics provide dtrace is a mechanism within Solaris that negotiates and monitor an almost infinite variety of data that is being processed or held within the system. You can take measurements at various points and the Storage Appliance allows you to easily monitor what’s going on to a very high degree and producing nice-looking graphs at the same time.

 

[pause]

 

As I mentioned, there’s always a new feature coming along. One of the latest ones is something that was invented or supplied with the Exadata systems. That was Oracle’s all-in-one database machine that’s called Hybrid Columnar Compression, which is a very advanced compression facility that gives you amazing amounts of compression and storage of data with a very high performance. Apparently, according to [4:43 inaudible] from Oracle, you get something like three to five times compression than anything the competition can achieve.

 

Copyright SkillBuilders.com 2017

×
Transcript

ZFS Storage Appliance Models

ZFS Storage Appliances – Models (Part 3 of 6)

 

>> Mick:  There are three basic model ranges – the 7120, 7320, and 7420 – the 7420 being the top of the food chain as it were, which can store up to 1.15 petabytes of data. Starting with the 7120, anything from 20 terabytes up to the physical capacity of that system.

 

Each model can be expanded with additional disk shelves, additional flash storage. Obviously, it can be configured in the first place with the correct type of disks that you need to be using. As you can see here, the 7120 uses 7200 rpm disks, while the 7320 and 7420 can also use 15K SAS disks.

 

Also the higher end models can be configured in clusters failover. Some very advanced features there. Every protocol that you need for file sharing is present as we’ll see during the demonstration.

 

The system upgrade expose both as a nasty [1:22 inaudible] in network attached storage and the [1:25 inaudible]. So you can do FibreChannel and you can do NFS, whatever you need. And you can add extra devices for input/output. By default, the system has four 1 gig Ethernet interfaces which is pretty standard on most Oracle servers and appliances, but you can add another quad card, gigabit card, Infinibands, 10 gig Ethernet, which I’ve lots of mention here and also a FibreChannel HBOs. Obviously, the capacity options vary according to the model.

 

Copyright SkillBuilders.com 2017

×
Transcript

Introduction to the Demos

Oracle ZFS Storage Appliances – Intro to the Demos of ZFS Storage Appliances (Part 4 of 6)

 

>> Mick:  What we’re going to do now then is demonstrate the system – demonstrate the software. I’ve got a machine here.

 

[pause]

 

I’m actually running a virtual machine in the Oracle VM VirtualBox system. You can download the ready-made ZFS Storage Appliance Server with a whole bunch of emulated disk drives, which is what I’ll be doing.

 

[pause]

 

But bear in mind, although I’m not really using the actual Storage Appliance, what I’m doing here is exactly the same as I would on many actual physical appliance. There’s no difference between what I’m going to show you and what actually happens on a physical machine.

 

We’re going to look at the initial power and configuration at which this thing is sitting, waiting for me to start. Part of the configuration requires knowledge of how we want the disks to be used. Do we want to use something like RAID 5? Do we want RAID 1, which is mirroring, and so forth? You’ll see how we have to decide. Using the different RAID levels, how we want to use the disks, which might depend a little bit on the load that we’re going to be using on this system.

 

Then we’re going to look at creating shares, including NFS and CIFS which is Microsoft file share. I’m going to be doing things like setting up a user account for demonstration purposes, a project. Project to use in the Storage Appliance to allow us to categorize the shares we want so when we have many shares and many users, projects will allow us to keep a handle on what’s going on.

 

You can also do things like iSCSCI which is the small computer interface (SCSCI disks) but using the network rather than the data cable. When you create an iSCSCI share it will be seen on your server as though it was a physical disk, so the Storage Appliance can do all the latest technology.

 

[pause]

 

It obviously has lots of management facilities too and if you already have a database of users that you want to use and DNS servers then you can point the Storage Appliance at your active directory server, for example, or local DNS server. And you can authenticate users through an active directory, for example.

 

As Dave mentioned earlier, we’re expecting that you already know a little bit about the storage technology that we’re going to be talking about.

 

[pause]

 

As I mentioned, managing the software is the same across all models. We’re actually running a simulator inside VirtualBox. If you’re interested, you can get VirtualBox under Windows, Linux, and Solaris but only on Intel hardware but not SPARC. And the VM image for the Storage Appliance for which we will provide a link, can be freely downloaded from Oracle anytime.

 

[pause]

 

Just a note about SkillBuilders to let you know that we are involved in this technology by the day and we’ve got the resources, we have the certifications and the experience to help you evaluate the products, help you install them and help you maintain them on a day to day basis. That includes not only the Solaris operating system and the Storage Appliance system but also some great skills in the Oracle database itself and related technologies. If you’re interested then give us a call. There’s a number for you.

 

Copyright SkillBuilders.com 2017

×
Transcript

Demonstration – Configuring a ZFS Storage Appliance

Oracle ZFS Storage Appliances – Demonstration Configuring a ZFS Storage Appliances (Part 5 of 6)

 

>> Mick:  We’re going to begin our demo now. We’ll switch to the VirtualBox Appliance here.

 

[pause]

 

I’ve effectively installed the box, connected to it from the system console and switched it on. It’s booted. If you watch carefully, you’ve probably seen that it is in fact really a Solaris system and it’s waiting for me to begin configuration.

 

If we press any key to start – and I’ll have to fill in the basic configuration here as you can see. The reason that configuration already exists is because the system uses the dynamic host configuration protocol to go off and get network details which I want to change to be static. So I’m going to change the host in there.

 

[pause]

 

The system actually has a very useful command line interface as well as a nice browser based interface. Although most of the rest of the demo will be via the browser I would hopefully spend a minute or two showing you commands that I can use from the command line as well. In fact, you can even do scripting if you’re doing lots of repetitive commands each step. Just like any computer system, you can create scripts to have them executed.

 

[pause]

 

All the other details here are okay. I’ll enter a password and then esc-1 which tells the system I’ve finished entering the configuration data.

 

There we go. The system’s come back to me now and it’s telling me that I should talk to the appliance on the network address I’m assigned and then port number 215. I could log in here on the console and I could do things here at the command line but it’s a bit nicer to show it through the basic user interface. Certainly the early stages of the configuration are also a little bit easier doing that.

 

So I’ll start up a web browser and I’ll go into the interface which my system rightfully doesn’t trust. I’ll proceed anyway. And there we go. I’m connected up to the Storage Appliance. Log in as root, give the password.

 

[pause]

 

The system realizes that this is the first time that machine has been used and I need to do a little bit further configuration before I can get started.

 

I click on start and it will take me through a half a dozen steps to get the thing up and running. First of which is to configure networking. Bear in mind that this is initial configuration and if I make a mistake or change my mind or I don’t really want to do any configuration at the moment, I can always come back a little bit later and do this.

 

The thing has got two built-in Ethernet interfaces. A normal physical machine we have four. One of which I configured in the early stages of configuration, which you saw earlier. There’s nothing to change here really other than I might want to make these things a little bit less anonymous.

 

[pause]

 

For example here, I’ve got my data link which is my physical link and here I’ve got my interface which is the link with the IP address reside, etc.

 

Imagine I wanted to edit this. If you point to the object can you see there’s a trash can symbol so I could remove the information or there’s a little pen which I could click on to make changes to edit. The name of the host is hoki and I want to call it hoki datalink because I’m going to use that interface as a published interface for people access the shares on this device and reserve the other one for management purposes.

 

Notice while we’re on this form that I can actually use it to create link aggregations. A link aggregation is fairly sophisticated mechanism for failover and load spreading, so I can combine a number of Ethernet interfaces into one apparent physical interface. LACP aggregation then would ask me if I clicked on that to enter some information about the link aggregation.

 

The LACP is link aggregation control protocol and once the aggregation is up and running, the protocol will be used to inform the switch that the link aggregation exists and hopefully the switch will configure itself accordingly. It is worth noting that you need something like a layer 3 switch or certainly a network switch that supports link aggregations, which I don’t want to do. All I need to do is change the datalink. But I can also set up a VLAN.

 

[pause]

 

My interface also – it just started up a little bit – I’ll call it hoki data-yf and at this layer of the protocol note I can use IP multipathing if I want to. Multipathing is not quite sophisticated as link aggregations, but it’s a very well used technology in Solaris and it provides failover and a certain degree of network load spreading. At the moment I don’t want to use that so I’ll bypass it. Just change the name. Once I’ve completed this step, just click on commit.

 

[pause]

 

I didn’t apply so it’s warning me. Now I must know my DNS domain which I actually already put in at the early configuration stage.

 

Manny wants to know NTP, which is network time protocol. I can give it some machine name or IP addresses to go and look at to configure the network time protocol, which is very important for a system like this to make sure if it’s a storage device then the time stamps, etc. on the files that you’re recording are absolutely correct. I’m going to use local ones here.

 

[pause]

 

It’s always best to have two or three at these machines.

 

[pause]

 

Two will do us today. The more the merrier really, because the system will go to each of these machines and work out a time that it can deduce from both of them.

 

We’ll sync our clock and then we’ll commit. If there’s any problem contacting these systems, we’ll get an error. This will validate the path where they can access the NTP servers. We’re actually doing it, so I’m keeping my fingers crossed.

 

Now it wants to know name services because they could be storing user account details on some remote server. I may already have a number of user account setup and I can choose a variety of different name services. A name service is basically a store of administration information, perhaps IP name and addresses which specifically be DNS or usernames, passwords, IDs, group, name and password IDs, and a whole variety of other administration that will be very useful to store centrally.

 

Quite common ones these days is Active Directory which is a Microsoft mechanism. This client can authenticate through – if you as a user try and access a share on here then the system can go off to an active directory box and say, “Is this a valid user?” We don’t want any of these things at the moment.

 

Now we have to configure the actual hardware.

 

[pause]

 

You’ll see a number of screens as you use this software. You’ll see things like available pools with a plus (+) and you click on the plus (+) and you can start adding things. You have to give the ZFS pool a name. The pool will be an area of space that is given to us by assigning a number of disks to it. It’s sort of a logical [9:50 inaudible], if you like. I’m going to call it some pool name. I’ll use superior which is that very large pool in North America and obviously a good description of the product. The two things go well together.

 

[pause]

 

I’ve got 15 disks on this appliance. Remember it’s a virtual appliance so the capacity is actually quite small. But for demo purposes, it’s quite good. I might create a pool using all the disks depending on how I’m going to configure them shortly or I could create different pools for different system tiers.

 

Obviously, if you’re interest into making sure the management is simple then just create a single pool. I’ll choose a number of disks fairly randomly. Obviously, you normally think about this a bit and commit to it.

 

[pause]

 

This screen is asking me for the disk storage I’ve selected, what type of data profile do I want to use? Do I want it mirrored, single parity RAID, double parity RAID, striped which doesn’t give us any redundancy but very high performance, or triple mirrored? And you can see the characteristics over here. Availability would be the redundancy so tripled mirrored would be very much available as with double parity RAID. Mirrored also provides good redundancy and also a very good performance.

 

There is obviously some thought required to work out exactly – for the workload you got, what would be the most suitable? I don’t want to spend too much time looking at these things now because I’m more interested in showing you the general purpose facilities of the device. So I’m going to choose mirrored.

 

[pause]

 

I’ll commit.

 

[pause]

 

There’s the summary of my data profile with my single pool.

 

>> Dave:  Mick, to this pool notion, there is a question in the queue about should everything go on a single pool? Does it have to be in a single pool?

 

>> Mick:  No. You can have as many as you like, Dave. It’s just administratively, it works out to be easier if you have a single pool because then you have a totally transparent amount of space and within that space you can add individual shares, you can create different projects. So really, you don’t need more than one pool unless you want to have a completely different type of data profile there from one type of server to another. But usually speaking, you just have a single pool.

 

[pause]

 

>> Dave:  Perfect. Thanks, Mick.

 

[pause]

 

>> Mick:  Okay. The GUI is now allowing us to register online, which we don’t need to do at the moment.

 

Copyright SkillBuilders.com 2017

×
Transcript

Demonstration – ZFS Storage Appliance Details

Oracle ZFS Storage Appliances – Demonstration  ZFS Storage Appliance Details (Part 6 of 6)

 

>> Mick:  Now we’re going to look at the Storage Appliance in some detail.

 

[pause]

 

The first thing we’re going to do is just have a quick look through the different menus that you see here at the top. You can see that within the different menus – can you see here you have services, storage, network, etc. on the configuration? These are maintenance menu which allows you to do things with the system.

 

For example, do a reset which is what I have to do every time I want to start from scratch for example. I can create shares. I can look at the system status and my status screen showing various analytics or I can generate specific analytics.

 

[pause]

 

Just let’s get started by showing you how I can add statistic, say a percent utilization of the CPU and instantly I’ll get a nice graph coming up. I can add another statistic like network interface statistics. I can go on like this creating whatever statistics I like. Using these little buttons here, these icons, I can show a minute of data, show an hour. Obviously, the appliance would need to be running for some time before I could actually show these in full detail – show a full day, etc. I can represent data by different types of graph. You can see pretty detailed information is available.

 

[pause]

 

Let’s go back. What I’ll do is start by going into configuration and I’ll create a user account. The analogy would be just like creating a user on a Microsoft system or Unix box. I’ll click on plus (+), I’ll create a user. ZFSuser1, local only, not on the network directory. Clear the password.

 

[pause]

 

The things here require sessionalization. It would mean if they user logged in, they have to type a reason as to why they’re logging in, if they have some sort of admin function on the Storage Appliance. I only want to use the user for authenticating when I access the share. If I click kiosk user, that would then restrict what they could see when they use the browser interface to look at the Storage Appliance.

 

[pause]

 

I’m going to add the user then I’m going to go into shares. Rather than create a single share, which I could do, I’m going to go into projects so I can categorize my share under project which will make it easier for me to manage a bit later. I’m going to add a project called SkillBuilders.

 

[pause]

 

Now I’m going to go back into shares and bring up my projects, select SkillBuilders. Now when I create a file system that will be associated with that project (as you can see) and I’m going to create a share called reports owned by the user I’ve just created, zfsuser1. Here I can set my permissions, which you can see are Unix permission groups, and I can click on individual permissions. I’m just going to leave read-write execute for owner at the moment.

 

[pause]

 

From there I’ve created a share. Now you might say, “How would you see this?” Just like any other share. If I go into a remote session, I’m going to log into a local system, logging in as zfsuser1. I’ve got the equivalent user account setup on my Solaris 10 system.

 

[pause]

 

If I do showmount-c, I can see the report share is exported. Note the default mount point export report which I can change if I want to but everything is shared from /export.

 

In order to mount this manually I would need to become root on my Solaris box and then mount mnt directory and /mnt. As you may be aware, if you’re used to Solaris, it is always a mnt directory that’s handy for mounting stuff. Usually it is.

 

[pause]

 

Maybe I’ve already got it mounted. Let’s make a different directory. That’s better.

 

[pause]

 

>> Dave:  A couple of questions in the queue, Mick. Is this a good time?

 

[pause]

 

>> Mick:  Fire away, Dave.

 

[pause]

 

>> Dave:  You’re using the command line now and as far as the configuration that you did before, can command line be used for the ZFS Storage Appliance?

 

>> Mick:  Yes. The command line can be used under any circumstances that I’m showing you graphical facilities for. In fact, the command line has more capability than the browser interface.

 

In this particular window here – I’m actually logged into a different system so I’m actually accessing the share from a Solaris machine. As you can see, I have mounted it but I cd to it, even though I’m root, it won’t allow me. If I drop back to my ZFS user and then cd, I’m in because that was the user I used to identify the system when I created the report share.

 

Just to show you hopefully I can make a directory – and there we are. I could always start adding data but because it’s an emulated system, the performance isn’t particularly good. But you can see that I can indeed share and I can access the share and I can control access to it for user, and so forth.

 

[pause]

 

If I go back to shares – there’s my share that I’ve created – it’s important to note, imagine I want to edit that and I can’t see an edit button anywhere. If I point to it, notice how you can now see the edit. So I can go in and I can change aspects, I could add a quota which is a limitation of the amount of space it could use. A reservation is a guarantee of a minimum amount of space which is ZFS facility in its own right.

 

[pause]

 

I could add up the facilities to that share, make data compression.

 

[pause]

 

I choose from a variety of compression characteristics and a variety of other properties at the same time. Then click “Apply” to have those take effect.

 

Lots of properties you can apply or you can just leave everything to inherit from the main project directory. Again, this is a facility of ZFS where you create within your pool when you’re creating a share like this. You’re effectively creating what’s known as a dataset which you can view a separate file system, which in itself has a whole variety of properties assigned to it such that if you then create other things within it, those datasets inherit the properties of the parent which is what you’re doing with one, which is a very powerful facility. I can do things like take snapshots.

 

[pause]

 

Just a snaphot. Give it a name.

 

[pause]

 

They are taking the snapshot just like that of that report share, which means anything that is now created we’ll probably see being reflected in the file system, but if I change my mind later or something goes wrong, I can revert back to this one and so on. Bear in mind, I’m doing this through the graphical interface and in situations like this where I might have multiple file systems, I would probably be better off using the command line interface and the script to do things like snapshotting. Replication…

 

[pause]

 

I can do a variety of replications which somebody was asking about just now.

 

[pause]

 

If we go back to our share here…

 

[pause]

 

Start copying the data across to the share and then we have a look at analytics. We should be able to see – there we go – some activity starting to happen.

 

[pause]

 

Not an awful lot because it didn’t copy very much, but at least you can see it.

 

>> Dave:  Mick, any scripting capabilities?

 

[pause]

 

>> Mick:  Yes, there are, from the command line, there are scripting capabilities. Certainly, yes. Any commands that you might input at the command line, you can script them so you can do repetitive things. There are control statements just like in [11:45 inaudible] program and language that you can use, as well.

 

[pause]

 

A thing about the shares – one other thing I’d like to show you in the last few minutes is if we’d look at configuration and the services that are currently available, you can see the NFS and SCSI are actually switched on. Currently, we only have NFS share. We don’t have an SCSI one. But we might like to create a Microsoft share that I could access from my laptop. The SMB service isn’t currently switched on, so let’s click on it.

 

[pause]

 

This is just the generic service that is built in to the Solaris system.

 

[pause]

 

And when we click on “Apply,” it says, “You made changes,” and “It’s currently disabled. Would you like to enable it?” I’ll enable it.

 

[pause]

 

Let’s go back to shares.

 

[pause]

 

Go back to SkillBuilders project.

 

[pause]

 

Set up a new share.

 

[pause]

 

We’ll use our original user account when we set up, zfsuser1. Click on “Apply.” We now have our two shares. If we go back to our Window system here, go into network and map the network drive.

 

[pause]

 

Here we go. I’m on a Window 7 system, by the way. I can obviously choose drive letter for which the share to be associated.

 

[pause]

 

I won’t be too optimistic and use the machine name although it should work quite happily. I’ll use the IP address.

 

[pause]

 

We’ll connect using different credentials. That is different credentials to the user I’m currently logged in as on my laptop.

 

[pause]

 

Check “Remember.” Click on “Okay.” And it’s now trying to connect to the Storage Appliance.

 

If we go back to the analytics again, we can see the network and the CPU activity going on within the machine. We could add further statistics for disk performance, etc.

 

[pause]

 

>> Dave:  Another question in the queue, Mick.

 

>> Mick:  Yeah.

 

>> Dave:  Why is rollback time slow? Are there ways to speed the rollback of a large LUN?

 

[pause]

 

>> Mick:  I’m not aware that it is slow, Dave.

 

>> Dave:  I’ll ask for more clarification.

 

>> Mick:  Yeah, okay.

 

[pause]

 

Here we can see the various statistics being shown. Don’t forget, you can always look at the status screen to find out what’s going on. Now one last thing very quickly is I can log in on the command line and I get a prompt. It’s not a shallow prompt. If I hit “Tab,” conveniently it shows the all the different things I can do. So I can come into shares and then show. Hit “Tab” and then shows me what I can do within shares. That is the script option.

 

[pause]

 

Cd.. will get back up a little. Everything I’m doing through the GUI I can also do through the command line. In actual fact, when it comes to things about scripting, I can do more through the command line than I can do with the browser user interface.

 

That’s the end of the demonstration. I hope that’s given you a bit of an idea of what the software can do and also I’ve given you a bit of an overview of what the hardware options are.

 

Thank you very much for attending. Let us know if there are any more questions that we can get back to you. Thanks once again and happy Christmas. Over to you, Dave.

 

>> Dave:  Thank you so much, Mick. Fantastic demonstration and presentation. I appreciate it very much.

 

[pause]

 

Wishing everybody a happy holiday time. There’s an outstanding question in the queue about that rollback. I’d like for you to email that to us, Dave@Skillbuilders.com is just a great way to contact me. Dave@Skillbuilders.com, I’ll put it in the chat window here and we’ll get to the bottom of that. So if you have any further questions, folks, you can contact us via email or the phone number which I’ll put in the chat window as well. Everyone, have a happy Christmas time and happy holiday time. Thanks for attending today’s webinar. Thanks again, Mick, it was a great demonstration.

 

>> Mick:  Thanks, Dave. Thanks, everybody. Bye for now.

 

Copyright SkillBuilders.com 2017

×