| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-15 | |||
| 13:56:47 | artom | sean-k-mooney, you mean https://review.opendev.org/#/c/667177/? | |
| 13:56:51 | sean-k-mooney | sorry https://review.opendev.org/#/c/663405/ is the tempest chage | |
| 13:56:59 | artom | Yeah, we thought it'd be fine | |
| 13:57:03 | artom | Apparently not | |
| 13:57:23 | artom | And like I said, I checked logs for 2 failing runs, everything seems OK from the Nova POV | |
| 13:57:26 | sean-k-mooney | right so https://review.opendev.org/#/c/667177/ is likely not enough | |
| 13:57:39 | artom | We correctly wait for plug-time events, we received them, we finish booting up the guest | |
| 13:57:40 | sean-k-mooney | it look like the neutron l3 agent has not correctly set up the floating ip | |
| 13:57:50 | sean-k-mooney | and that is why the ssh connection is not working | |
| 13:58:00 | artom | sean-k-mooney, ping is failing | |
| 13:58:15 | artom | But with no working fip we'd expect both ping and SSH to fail | |
| 13:58:16 | sean-k-mooney | the vm has got an ip from dhcp | |
| 13:58:34 | sean-k-mooney | so it has network connectivity | |
| 13:58:48 | sean-k-mooney | but since the ping/ssh are failing then that means its a floating ip issue | |
| 13:58:52 | artom | I'll start by filing a bug, we can start skipping that test again | |
| 13:59:03 | artom | And then work on a solution in its own time | |
| 14:00:02 | artom | Should I file it under Neutron then? | |
| 14:00:05 | sean-k-mooney | i think the neutorn folks are aware of this | |
| 14:00:36 | efried | alex_xu: when looking through the patches, is there anything specific you want me to be aware of etc? | |
| 14:00:37 | sean-k-mooney | am yes but i would check with them first | |
| 14:00:45 | sean-k-mooney | artom: this is the important bit http://logs.openstack.org/31/668631/7/check/tempest-slow-py3/a16d7d9/job-output.txt.gz#_2019-07-14_15_36_51_307956 | |
| 14:01:09 | sean-k-mooney | it sent a select to confim it recieved the ip to the dhcp server | |
| 14:01:25 | sean-k-mooney | then we try to ssh in and it fails | |
| 14:02:49 | sean-k-mooney | actull it connect a little later http://logs.openstack.org/31/668631/7/check/tempest-slow-py3/a16d7d9/job-output.txt.gz#_2019-07-14_15_36_51_311500 | |
| 14:06:58 | sean-k-mooney | artom: ok so its the ping after teh rever that is failing | |
| 14:07:14 | artom | sean-k-mooney, yeah. | |
| 14:07:29 | alex_xu | efried: I added a device manager, want to hear your thought, is it something we need, or I'm overcomplex it. | |
| 14:07:41 | artom | sean-k-mooney, so my kids woke up (late sleepers), I'll be doing dad stuff got a bit, I'll check back in once I'm at the office | |
| 14:07:44 | sean-k-mooney | the differenice i am seeing is we dont seem to be trying to ssh before we ping in the final case | |
| 14:08:00 | sean-k-mooney | artom: sure no worries | |
| 14:08:31 | yonglihe | sean-k-mooney, I still not totally got the unit test work for 'orphan cleanup'. | |
| 14:08:40 | yonglihe | but still working on that. | |
| 14:10:27 | artom | sean-k-mooney, I guess I'll file the bug for now, in Neutron. We can always change component later. | |
| 14:11:04 | yonglihe | 'Add server sub-resource topology API' is on run queue. I fixed the merge conflict. Unit test should pass, currently zuul failure seems not my fault: https://review.opendev.org/#/c/621476/. (seems bumping on image saving test case) | |
| 14:20:29 | yonglihe | It hit Bug 1737634: http://status.openstack.org/elastic-recheck/index.html#1737634 | |
| 14:20:29 | openstack | bug 1713163 in tempest "duplicate for #1737634 test_delete_saving_image fails because image hasn't transitioned to SAVING" [Medium,Confirmed] https://launchpad.net/bugs/1713163 | |
| 15:03:08 | efried | dansmith: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-07-11.log.html#t2019-07-11T13:39:31 | |
| 15:03:08 | efried | LibvirtDriver.power_off() calls self._destroy() which calls guest.poweroff() which calls self._domain.destroy() | |
| 15:04:01 | efried | That is, unless I'm missing something, the domain XML is destroyed at power off, not during the startup. | |
| 15:05:56 | dansmith | efried: I'm not sure what you're saying or asking | |
| 15:06:25 | efried | dansmith: When we were talking about doing device (specifically vpmem) claims via the virt driver | |
| 15:06:33 | dansmith | efried: when we go into a reboot operation, before we do anything to the guest, we can look at the existing xml.. then if we need to un/redefine it, we can do that with the context we just learned | |
| 15:07:06 | efried | dansmith: Okay, but if I do a stop, then "wait a while", then start again, I've lost that information. | |
| 15:07:10 | dansmith | no | |
| 15:07:19 | dansmith | you do not lose the xml across a stop/start | |
| 15:07:43 | dansmith | a libvirt destroy operation does not lose data, it stops the instance (old xen terminology) | |
| 15:07:53 | dansmith | an undefine operation drops the stored definition of the guest | |
| 15:08:05 | efried | self._domain.destroy() <== does this lose the data? | |
| 15:08:07 | dansmith | no | |
| 15:08:18 | efried | oh | |
| 15:08:25 | efried | interesting choice of name | |
| 15:08:29 | dansmith | it comes from xen | |
| 15:08:33 | efried | alex_xu: ^ | |
| 15:10:41 | alex_xu | dansmith: why the start instance action is building xml... | |
| 15:11:11 | dansmith | alex_xu: to update the instance with any stored data or config changes that may have happened | |
| 15:11:31 | dansmith | since nova is the authority for a lot of things (like config drive or nic mac address, etc) | |
| 15:12:20 | alex_xu | ah.. | |
| 15:12:36 | efried | so why do we need to store things like PCI devs, NUMA topo, etc. in the db at all? | |
| 15:13:44 | dansmith | because we have apis for them? | |
| 15:14:04 | sean-k-mooney | efried: because we need to asign specfiic devices,cpus to instances and we dont persitit the libvirt xml | |
| 15:14:07 | dansmith | because we need aggregated views of that data without contacting every node in the cluste for each sort of thing/ | |
| 15:14:26 | sean-k-mooney | so if we dont store it in the db we dont know what ones are used if an instance is not running | |
| 15:14:53 | alex_xu | sean-k-mooney: we just talk about the libvirt actually persistent everything | |
| 15:15:01 | sean-k-mooney | also we need it for the filters | |
| 15:15:20 | sean-k-mooney | alex_xu: that would cause other issues on upgrade of libvirt versions and nova | |
| 15:15:20 | dansmith | sean-k-mooney: not sure that's true (that we can't get it from libvirt from stored domains), | |
| 15:15:46 | dansmith | but the operations we need to do would be incredibly expensive if we had to collect "who is using what right now" by contacting every node | |
| 15:16:14 | sean-k-mooney | dansmith: well if we delete teh domain on power off we need to reserve the cores so that we dont violate the requiremetn by booting a new vm | |
| 15:16:15 | efried | operations like scheduling, determining available resources... | |
| 15:16:23 | sean-k-mooney | if we dont undefine the domain on power off | |
| 15:16:31 | dansmith | sean-k-mooney: again, we do not delete the domain on poweroff | |
| 15:16:32 | sean-k-mooney | then its possible we dont need to store tehm for that | |
| 15:16:40 | sean-k-mooney | but we would still want to have it for the filter | |
| 15:17:09 | sean-k-mooney | we do on power on because it calls hard reboot | |
| 15:17:14 | efried | So in the utopian future where all of that is done properly in placement, the scheduler wouldn't need that info in the db and we could theoretically get rid of it. | |
| 15:17:15 | dansmith | right, we need that info stored because we've promised a cluster-wide API for them, and because the filters need to be able to survey the entire landscape efficiently | |
| 15:17:21 | sean-k-mooney | we might not on power off i didnt check | |
| 15:17:41 | sean-k-mooney | e.g. ill take your word for it | |
| 15:20:17 | efried | dansmith: so, for a new resource we're tracking, specifically vpmem for this conversation, | |
| 15:20:17 | efried | since we're tracking the inventory properly in placement, | |
| 15:20:17 | efried | that means only the virt driver needs to know the connection between an allocation and a specific vpmem namespace, | |
| 15:20:17 | efried | so there's no reason to store vpmems in the database | |
| 15:21:09 | dansmith | if that is doable (i.e. we don't have to store them for some other reason) then I think that's ideal, yes | |
| 15:21:13 | efried | and, | |
| 15:21:13 | efried | since we can recover the vpmem information from the domain xml on operations other than migrations (see below), there's no need to store the vpmem info on the Instance either. | |
| 15:21:29 | efried | So the only place we actually need the information is in a migrate context | |
| 15:21:50 | dansmith | do we need it in a migration context? | |
| 15:22:03 | alex_xu | yes | |
| 15:22:06 | efried | not sure | |
| 15:22:07 | dansmith | the two ends of a migration are storing the machine-local context in their own libvirts | |
| 15:22:22 | dansmith | maybe for the revert case? | |
| 15:22:23 | efried | alex_xu: why would we need it in migration context? | |
| 15:22:24 | alex_xu | I need copy the source pmem data to the dest pmem. I need dest pmem device path | |
| 15:22:46 | dansmith | wait | |
| 15:22:58 | dansmith | so for cold migration you're going to migrate the data as well? | |
| 15:23:05 | alex_xu | yes | |
| 15:23:11 | efried | there's a conf option | |
| 15:23:25 | alex_xu | yes, it is configurable by the extra spec | |
| 15:23:31 | dansmith | how is that going to work? write it to disk, consider it a disk image to move and then write it back? | |
| 15:23:41 | sean-k-mooney | dansmith: not by defualt | |