| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-15 | |||
| 13:35:55 | sean-k-mooney | sorry not per line but per log output | |
| 13:36:30 | sean-k-mooney | e.g. if you log multi line message teh log/parser/setream will stream that full log messge to logstash in one go | |
| 13:38:11 | artom | sean-k-mooney, oh, I think it splits on the _ | |
| 13:41:30 | sean-k-mooney | its shouldnt | |
| 13:41:44 | sean-k-mooney | it might be but it shouldn't | |
| 13:41:56 | alex_xu | efried: looking for some feedback on this direction https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:master+topic:claim_for_instance before I'm going futher. | |
| 13:41:59 | efried | artom: (still) not paying a lot of attention here, but there ought to be lots of good examples in the elastic-recheck project | |
| 13:42:16 | artom | efried, ack, thanks | |
| 13:42:43 | efried | alex_xu: Okay, I saw a bunch of patches come in this morning. Were you planning to put up a spec for this? | |
| 13:42:44 | sean-k-mooney | ya https://github.com/openstack-infra/elastic-recheck/tree/master/queries i was looking at those | |
| 13:42:49 | sean-k-mooney | artom: ^ | |
| 13:44:12 | artom | sean-k-mooney, yeah... and I appear to be doing everything right :( | |
| 13:45:13 | sean-k-mooney | i dont see any example of actully match on the tempest output | |
| 13:45:14 | artom | Wait, do I need to capitalize AND? | |
| 13:45:24 | sean-k-mooney | maybe | |
| 13:45:44 | sean-k-mooney | yes | |
| 13:46:42 | artom | *facepalm* | |
| 13:46:48 | artom | OK, now it's turning up nothing | |
| 13:47:14 | sean-k-mooney | artom: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration_revert%5C%22%20AND%20message%3A%5C%22FAILED%5C%22%20AND%20tags%3A%5C%22job-output.txt%5C%22 | |
| 13:47:19 | sean-k-mooney | set it to 30days | |
| 13:47:24 | sean-k-mooney | and things show up | |
| 13:48:25 | sean-k-mooney | the last failure was about an hour ago | |
| 13:49:10 | sean-k-mooney | looks like it start to show up on teh 13th | |
| 13:50:24 | artom | sean-k-mooney, that sounds about right | |
| 13:50:56 | artom | Means https://review.opendev.org/#/c/663405/ is causing it | |
| 13:51:47 | stephenfin | melwitt: If you're around today, could you take a look at this doc fix? https://review.opendev.org/#/c/670125/ | |
| 13:52:04 | artom | All on different changes, too | |
| 13:52:19 | sean-k-mooney | the first failure was https://review.opendev.org/#/c/627765/ | |
| 13:53:23 | sean-k-mooney | it might be caused by https://review.opendev.org/#/c/663405/ but it looks like a neutron issue to me | |
| 13:55:01 | artom | sean-k-mooney, it has to be "cause" by https://review.opendev.org/#/c/663405/ because before that landed we just didn't run that test :) | |
| 13:55:51 | alex_xu | efried: if we need a spec, I can write up one | |
| 13:56:00 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Integrate 'pre-commit' https://review.opendev.org/665518 | |
| 13:56:02 | sean-k-mooney | well we dont know that https://review.opendev.org/#/c/663405/ would fix the issue in that test | |
| 13:56:22 | sean-k-mooney | we speculated that it should | |
| 13:56:47 | artom | sean-k-mooney, you mean https://review.opendev.org/#/c/667177/? | |
| 13:56:51 | sean-k-mooney | sorry https://review.opendev.org/#/c/663405/ is the tempest chage | |
| 13:56:59 | artom | Yeah, we thought it'd be fine | |
| 13:57:03 | artom | Apparently not | |
| 13:57:23 | artom | And like I said, I checked logs for 2 failing runs, everything seems OK from the Nova POV | |
| 13:57:26 | sean-k-mooney | right so https://review.opendev.org/#/c/667177/ is likely not enough | |
| 13:57:39 | artom | We correctly wait for plug-time events, we received them, we finish booting up the guest | |
| 13:57:40 | sean-k-mooney | it look like the neutron l3 agent has not correctly set up the floating ip | |
| 13:57:50 | sean-k-mooney | and that is why the ssh connection is not working | |
| 13:58:00 | artom | sean-k-mooney, ping is failing | |
| 13:58:15 | artom | But with no working fip we'd expect both ping and SSH to fail | |
| 13:58:16 | sean-k-mooney | the vm has got an ip from dhcp | |
| 13:58:34 | sean-k-mooney | so it has network connectivity | |
| 13:58:48 | sean-k-mooney | but since the ping/ssh are failing then that means its a floating ip issue | |
| 13:58:52 | artom | I'll start by filing a bug, we can start skipping that test again | |
| 13:59:03 | artom | And then work on a solution in its own time | |
| 14:00:02 | artom | Should I file it under Neutron then? | |
| 14:00:05 | sean-k-mooney | i think the neutorn folks are aware of this | |
| 14:00:36 | efried | alex_xu: when looking through the patches, is there anything specific you want me to be aware of etc? | |
| 14:00:37 | sean-k-mooney | am yes but i would check with them first | |
| 14:00:45 | sean-k-mooney | artom: this is the important bit http://logs.openstack.org/31/668631/7/check/tempest-slow-py3/a16d7d9/job-output.txt.gz#_2019-07-14_15_36_51_307956 | |
| 14:01:09 | sean-k-mooney | it sent a select to confim it recieved the ip to the dhcp server | |
| 14:01:25 | sean-k-mooney | then we try to ssh in and it fails | |
| 14:02:49 | sean-k-mooney | actull it connect a little later http://logs.openstack.org/31/668631/7/check/tempest-slow-py3/a16d7d9/job-output.txt.gz#_2019-07-14_15_36_51_311500 | |
| 14:06:58 | sean-k-mooney | artom: ok so its the ping after teh rever that is failing | |
| 14:07:14 | artom | sean-k-mooney, yeah. | |
| 14:07:29 | alex_xu | efried: I added a device manager, want to hear your thought, is it something we need, or I'm overcomplex it. | |
| 14:07:41 | artom | sean-k-mooney, so my kids woke up (late sleepers), I'll be doing dad stuff got a bit, I'll check back in once I'm at the office | |
| 14:07:44 | sean-k-mooney | the differenice i am seeing is we dont seem to be trying to ssh before we ping in the final case | |
| 14:08:00 | sean-k-mooney | artom: sure no worries | |
| 14:08:31 | yonglihe | sean-k-mooney, I still not totally got the unit test work for 'orphan cleanup'. | |
| 14:08:40 | yonglihe | but still working on that. | |
| 14:10:27 | artom | sean-k-mooney, I guess I'll file the bug for now, in Neutron. We can always change component later. | |
| 14:11:04 | yonglihe | 'Add server sub-resource topology API' is on run queue. I fixed the merge conflict. Unit test should pass, currently zuul failure seems not my fault: https://review.opendev.org/#/c/621476/. (seems bumping on image saving test case) | |
| 14:20:29 | yonglihe | It hit Bug 1737634: http://status.openstack.org/elastic-recheck/index.html#1737634 | |
| 14:20:29 | openstack | bug 1713163 in tempest "duplicate for #1737634 test_delete_saving_image fails because image hasn't transitioned to SAVING" [Medium,Confirmed] https://launchpad.net/bugs/1713163 | |
| 15:03:08 | efried | dansmith: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-07-11.log.html#t2019-07-11T13:39:31 | |
| 15:03:08 | efried | LibvirtDriver.power_off() calls self._destroy() which calls guest.poweroff() which calls self._domain.destroy() | |
| 15:04:01 | efried | That is, unless I'm missing something, the domain XML is destroyed at power off, not during the startup. | |
| 15:05:56 | dansmith | efried: I'm not sure what you're saying or asking | |
| 15:06:25 | efried | dansmith: When we were talking about doing device (specifically vpmem) claims via the virt driver | |
| 15:06:33 | dansmith | efried: when we go into a reboot operation, before we do anything to the guest, we can look at the existing xml.. then if we need to un/redefine it, we can do that with the context we just learned | |
| 15:07:06 | efried | dansmith: Okay, but if I do a stop, then "wait a while", then start again, I've lost that information. | |
| 15:07:10 | dansmith | no | |
| 15:07:19 | dansmith | you do not lose the xml across a stop/start | |
| 15:07:43 | dansmith | a libvirt destroy operation does not lose data, it stops the instance (old xen terminology) | |
| 15:07:53 | dansmith | an undefine operation drops the stored definition of the guest | |
| 15:08:05 | efried | self._domain.destroy() <== does this lose the data? | |
| 15:08:07 | dansmith | no | |
| 15:08:18 | efried | oh | |
| 15:08:25 | efried | interesting choice of name | |
| 15:08:29 | dansmith | it comes from xen | |
| 15:08:33 | efried | alex_xu: ^ | |
| 15:10:41 | alex_xu | dansmith: why the start instance action is building xml... | |
| 15:11:11 | dansmith | alex_xu: to update the instance with any stored data or config changes that may have happened | |
| 15:11:31 | dansmith | since nova is the authority for a lot of things (like config drive or nic mac address, etc) | |
| 15:12:20 | alex_xu | ah.. | |
| 15:12:36 | efried | so why do we need to store things like PCI devs, NUMA topo, etc. in the db at all? | |
| 15:13:44 | dansmith | because we have apis for them? | |
| 15:14:04 | sean-k-mooney | efried: because we need to asign specfiic devices,cpus to instances and we dont persitit the libvirt xml | |
| 15:14:07 | dansmith | because we need aggregated views of that data without contacting every node in the cluste for each sort of thing/ | |
| 15:14:26 | sean-k-mooney | so if we dont store it in the db we dont know what ones are used if an instance is not running | |
| 15:14:53 | alex_xu | sean-k-mooney: we just talk about the libvirt actually persistent everything | |
| 15:15:01 | sean-k-mooney | also we need it for the filters | |
| 15:15:20 | sean-k-mooney | alex_xu: that would cause other issues on upgrade of libvirt versions and nova | |
| 15:15:20 | dansmith | sean-k-mooney: not sure that's true (that we can't get it from libvirt from stored domains), | |