| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-15 | |||
| 14:06:58 | sean-k-mooney | artom: ok so its the ping after teh rever that is failing | |
| 14:07:14 | artom | sean-k-mooney, yeah. | |
| 14:07:29 | alex_xu | efried: I added a device manager, want to hear your thought, is it something we need, or I'm overcomplex it. | |
| 14:07:41 | artom | sean-k-mooney, so my kids woke up (late sleepers), I'll be doing dad stuff got a bit, I'll check back in once I'm at the office | |
| 14:07:44 | sean-k-mooney | the differenice i am seeing is we dont seem to be trying to ssh before we ping in the final case | |
| 14:08:00 | sean-k-mooney | artom: sure no worries | |
| 14:08:31 | yonglihe | sean-k-mooney, I still not totally got the unit test work for 'orphan cleanup'. | |
| 14:08:40 | yonglihe | but still working on that. | |
| 14:10:27 | artom | sean-k-mooney, I guess I'll file the bug for now, in Neutron. We can always change component later. | |
| 14:11:04 | yonglihe | 'Add server sub-resource topology API' is on run queue. I fixed the merge conflict. Unit test should pass, currently zuul failure seems not my fault: https://review.opendev.org/#/c/621476/. (seems bumping on image saving test case) | |
| 14:20:29 | yonglihe | It hit Bug 1737634: http://status.openstack.org/elastic-recheck/index.html#1737634 | |
| 14:20:29 | openstack | bug 1713163 in tempest "duplicate for #1737634 test_delete_saving_image fails because image hasn't transitioned to SAVING" [Medium,Confirmed] https://launchpad.net/bugs/1713163 | |
| 15:03:08 | efried | dansmith: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-07-11.log.html#t2019-07-11T13:39:31 | |
| 15:03:08 | efried | LibvirtDriver.power_off() calls self._destroy() which calls guest.poweroff() which calls self._domain.destroy() | |
| 15:04:01 | efried | That is, unless I'm missing something, the domain XML is destroyed at power off, not during the startup. | |
| 15:05:56 | dansmith | efried: I'm not sure what you're saying or asking | |
| 15:06:25 | efried | dansmith: When we were talking about doing device (specifically vpmem) claims via the virt driver | |
| 15:06:33 | dansmith | efried: when we go into a reboot operation, before we do anything to the guest, we can look at the existing xml.. then if we need to un/redefine it, we can do that with the context we just learned | |
| 15:07:06 | efried | dansmith: Okay, but if I do a stop, then "wait a while", then start again, I've lost that information. | |
| 15:07:10 | dansmith | no | |
| 15:07:19 | dansmith | you do not lose the xml across a stop/start | |
| 15:07:43 | dansmith | a libvirt destroy operation does not lose data, it stops the instance (old xen terminology) | |
| 15:07:53 | dansmith | an undefine operation drops the stored definition of the guest | |
| 15:08:05 | efried | self._domain.destroy() <== does this lose the data? | |
| 15:08:07 | dansmith | no | |
| 15:08:18 | efried | oh | |
| 15:08:25 | efried | interesting choice of name | |
| 15:08:29 | dansmith | it comes from xen | |
| 15:08:33 | efried | alex_xu: ^ | |
| 15:10:41 | alex_xu | dansmith: why the start instance action is building xml... | |
| 15:11:11 | dansmith | alex_xu: to update the instance with any stored data or config changes that may have happened | |
| 15:11:31 | dansmith | since nova is the authority for a lot of things (like config drive or nic mac address, etc) | |
| 15:12:20 | alex_xu | ah.. | |
| 15:12:36 | efried | so why do we need to store things like PCI devs, NUMA topo, etc. in the db at all? | |
| 15:13:44 | dansmith | because we have apis for them? | |
| 15:14:04 | sean-k-mooney | efried: because we need to asign specfiic devices,cpus to instances and we dont persitit the libvirt xml | |
| 15:14:07 | dansmith | because we need aggregated views of that data without contacting every node in the cluste for each sort of thing/ | |
| 15:14:26 | sean-k-mooney | so if we dont store it in the db we dont know what ones are used if an instance is not running | |
| 15:14:53 | alex_xu | sean-k-mooney: we just talk about the libvirt actually persistent everything | |
| 15:15:01 | sean-k-mooney | also we need it for the filters | |
| 15:15:20 | sean-k-mooney | alex_xu: that would cause other issues on upgrade of libvirt versions and nova | |
| 15:15:20 | dansmith | sean-k-mooney: not sure that's true (that we can't get it from libvirt from stored domains), | |
| 15:15:46 | dansmith | but the operations we need to do would be incredibly expensive if we had to collect "who is using what right now" by contacting every node | |
| 15:16:14 | sean-k-mooney | dansmith: well if we delete teh domain on power off we need to reserve the cores so that we dont violate the requiremetn by booting a new vm | |
| 15:16:15 | efried | operations like scheduling, determining available resources... | |
| 15:16:23 | sean-k-mooney | if we dont undefine the domain on power off | |
| 15:16:31 | dansmith | sean-k-mooney: again, we do not delete the domain on poweroff | |
| 15:16:32 | sean-k-mooney | then its possible we dont need to store tehm for that | |
| 15:16:40 | sean-k-mooney | but we would still want to have it for the filter | |
| 15:17:09 | sean-k-mooney | we do on power on because it calls hard reboot | |
| 15:17:14 | efried | So in the utopian future where all of that is done properly in placement, the scheduler wouldn't need that info in the db and we could theoretically get rid of it. | |
| 15:17:15 | dansmith | right, we need that info stored because we've promised a cluster-wide API for them, and because the filters need to be able to survey the entire landscape efficiently | |
| 15:17:21 | sean-k-mooney | we might not on power off i didnt check | |
| 15:17:41 | sean-k-mooney | e.g. ill take your word for it | |
| 15:20:17 | efried | dansmith: so, for a new resource we're tracking, specifically vpmem for this conversation, | |
| 15:20:17 | efried | since we're tracking the inventory properly in placement, | |
| 15:20:17 | efried | that means only the virt driver needs to know the connection between an allocation and a specific vpmem namespace, | |
| 15:20:17 | efried | so there's no reason to store vpmems in the database | |
| 15:21:09 | dansmith | if that is doable (i.e. we don't have to store them for some other reason) then I think that's ideal, yes | |
| 15:21:13 | efried | and, | |
| 15:21:13 | efried | since we can recover the vpmem information from the domain xml on operations other than migrations (see below), there's no need to store the vpmem info on the Instance either. | |
| 15:21:29 | efried | So the only place we actually need the information is in a migrate context | |
| 15:21:50 | dansmith | do we need it in a migration context? | |
| 15:22:03 | alex_xu | yes | |
| 15:22:06 | efried | not sure | |
| 15:22:07 | dansmith | the two ends of a migration are storing the machine-local context in their own libvirts | |
| 15:22:22 | dansmith | maybe for the revert case? | |
| 15:22:23 | efried | alex_xu: why would we need it in migration context? | |
| 15:22:24 | alex_xu | I need copy the source pmem data to the dest pmem. I need dest pmem device path | |
| 15:22:46 | dansmith | wait | |
| 15:22:58 | dansmith | so for cold migration you're going to migrate the data as well? | |
| 15:23:05 | alex_xu | yes | |
| 15:23:11 | efried | there's a conf option | |
| 15:23:25 | alex_xu | yes, it is configurable by the extra spec | |
| 15:23:31 | dansmith | how is that going to work? write it to disk, consider it a disk image to move and then write it back? | |
| 15:23:41 | sean-k-mooney | dansmith: not by defualt | |
| 15:23:44 | alex_xu | https://review.opendev.org/#/c/634556/12/nova/privsep/libvirt.py | |
| 15:23:54 | dansmith | I *love* this snowflake feature you guys have come up with | |
| 15:23:56 | sean-k-mooney | we are default to not copy the data but we optionally can copy it | |
| 15:23:58 | alex_xu | ^ here is, ssh... | |
| 15:24:17 | dansmith | alex_xu: so you're going to add another scp operation? | |
| 15:24:27 | alex_xu | yes... | |
| 15:24:34 | dansmith | gross | |
| 15:24:34 | alex_xu | it is something we hate... | |
| 15:24:59 | efried | What's the alternative? Block migrations etc. Which is worse? | |
| 15:25:04 | dansmith | so if/when we get to the point of being able to use libvirt to move the images over the TLS tunnel, we're stuck scp'ing this thing? | |
| 15:25:23 | dansmith | efried: not promising that this data is persistent across moves is one option | |
| 15:25:37 | dansmith | efried: isn't this being added as a snowflake "you can never live migrate" feature? | |
| 15:25:54 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: nova-manage: heal port allocations https://review.opendev.org/637955 | |
| 15:25:57 | efried | I thought live migrate was supported | |
| 15:25:58 | sean-k-mooney | dansmith: in the livemigration case qemu/libvirt can copy the data | |
| 15:26:01 | alex_xu | it support lm | |
| 15:26:23 | sean-k-mooney | for cold migrate we have to do it but maybe we can make a libvirt feature request? | |
| 15:26:35 | dansmith | okay, I thought it wasn't going to be able to move that data | |
| 15:26:44 | alex_xu | i'm ok without copy for resize | |
| 15:27:10 | efried | http://specs.openstack.org/openstack/nova-specs/specs/train/approved/virtual-persistent-memory.html#live-migration | |
| 15:27:11 | sean-k-mooney | alex_xu: the issue with scp/rsync for resize is it wont work for cross cell resize | |
| 15:27:22 | efried | apparently libvirt moves it along with RAM | |
| 15:27:33 | sean-k-mooney | alex_xu: e.g. for cross cell resize we cant assume teh compute nodes have network connectivtiy | |
| 15:27:41 | alex_xu | sean-k-mooney: yes, we are going to stop cross cell resize in the initial proposal | |