| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-22 | |||
| 07:20:53 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (14) https://review.opendev.org/576027 | |
| 07:21:05 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (15) https://review.opendev.org/576031 | |
| 07:21:16 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (16) https://review.opendev.org/576299 | |
| 07:21:33 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (17) https://review.opendev.org/576344 | |
| 07:21:51 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (18) https://review.opendev.org/576673 | |
| 07:22:10 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (19) https://review.opendev.org/576676 | |
| 07:22:26 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (20) https://review.opendev.org/576689 | |
| 07:22:38 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (21) https://review.opendev.org/576709 | |
| 07:22:50 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (22) https://review.opendev.org/576712 | |
| 07:23:08 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in virt/test_block_device.py https://review.opendev.org/566153 | |
| 07:23:37 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add a live migration regression test https://review.opendev.org/641200 | |
| 07:29:02 | openstackgerrit | zhongshengping proposed openstack/nova master: Replace git.openstack.org URLs with opendev.org URLs https://review.opendev.org/654398 | |
| 07:42:00 | boxiang | hi does anyone know why my nova-next fails http://logs.openstack.org/20/645520/11/check/nova-next/afbb160/? only nova-next of zuul fails. thanks :) | |
| 07:42:19 | boxiang | is it necessary for me to do 'recheck'? | |
| 07:58:30 | openstackgerrit | Boxiang Zhu proposed openstack/python-novaclient master: Add host and hypervisor_hostname to create servers https://review.opendev.org/647671 | |
| 11:45:34 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Updates for OpenDev transition https://review.opendev.org/654429 | |
| 13:26:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional recreate test for regression bug 1825537 https://review.opendev.org/654066 | |
| 13:26:47 | openstack | bug 1825537 in OpenStack Compute (nova) "finish_resize failures incorrectly revert allocations" [Medium,In progress] https://launchpad.net/bugs/1825537 - Assigned to Matt Riedemann (mriedem) | |
| 13:26:47 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop source node allocations if finish_resize fails https://review.opendev.org/654067 | |
| 13:29:49 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Drop py35 tests https://review.opendev.org/643873 | |
| 13:42:15 | mriedem | bauzas: can you review this functional recreate test for a stein regression bug? https://review.opendev.org/#/c/653268/ | |
| 13:45:24 | mriedem | jaypipes: thanks for looking at https://bugs.launchpad.net/nova/+bug/1825435 - replied to your oslo.config question | |
| 13:45:26 | openstack | Launchpad bug 1825435 in OpenStack Compute (nova) "TestRPC unit tests intermittently fail with "'>' not supported between instances of 'NoneType' and 'datetime.datetime'" - maybe due to "Fatal Python error: Cannot recover from stack overflow."" [High,Confirmed] | |
| 13:45:32 | openstackgerrit | Eric Fried proposed openstack/nova master: Fix {min|max}_version in ironic Adapter setup https://review.opendev.org/654457 | |
| 13:45:36 | efried | eandersson: ^ | |
| 13:46:02 | efried | mriedem: Just grinding into action here - any progress on that bug? | |
| 13:46:11 | mriedem | efried: no | |
| 13:46:17 | efried | k | |
| 13:46:21 | lpetrut | hey, I was wondering which criteria were used when defining the list of cpu traits: https://github.com/openstack/os-traits/blob/master/os_traits/hw/cpu/x86.py | |
| 13:46:30 | lpetrut | I mean, why not use all the cpu features defined here? https://github.com/libvirt/libvirt/blob/master/src/cpu_map/x86_features.xml | |
| 13:46:47 | mriedem | alex_xu: ^ | |
| 13:47:47 | jaypipes | mriedem: ack. still, the collections code does seem suspiciously related, given that stacktrace. | |
| 13:48:07 | mriedem | jaypipes: yeah i'm trying to find the python3.6 package change log for bionic | |
| 13:48:20 | mriedem | b/c i started seeing seg faults in pip3.6 last week also | |
| 13:48:24 | jaypipes | lpetrut: I added those x86 traits. | |
| 13:49:01 | jaypipes | lpetrut: I didn't use libvirt's XML files | |
| 13:49:34 | mriedem | hmm http://changelogs.ubuntu.com/changelogs/pool/main/p/python3.6/python3.6_3.6.7-1~18.04/changelog isn't very helpful | |
| 13:50:55 | lpetrut | got it, I was wondering how you've picked them, whether there was already some list at the libvirt driver level. also, would it make sense to extend this list? fetching all the features from the libvirt xml may be the easiest option | |
| 13:51:54 | jaypipes | lpetrut: there are more hypervisors than libvirt. :) | |
| 13:52:20 | jaypipes | lpetrut: and the content of the libvirt XML file changes over time of course. | |
| 13:52:57 | jaypipes | lpetrut: we've added things to os-traits as needed. if there's a particular instruction set extension you want/need, don't hesitate to throw up a patch adding it. :) | |
| 13:53:05 | lpetrut | jaypapipes: indeed, in fact I was looking into implementing 'update_provider_tree' for the hyper-v driver :) but those should probably be standardized at the openstack level | |
| 13:53:16 | jaypipes | ++ | |
| 13:53:49 | jaypipes | lpetrut: to answer your question, I don't remember there being any rhyme or reason to the ones I added. probably I just scraped things from unit tests or boxes I had lying around ;) | |
| 13:54:26 | lpetrut | got it, so basically each feature/trait was defined as needed (as opposed to having an extensive list) | |
| 13:55:14 | mriedem | lpetrut: do you guys plan on upstreaming feature parity stuff like this at some point? https://opendev.org/x/compute-hyperv/commit/d4f1fa457a0fcbd1ac84c816157175c090958d6f | |
| 13:55:30 | mriedem | https://opendev.org/x/compute-hyperv/commit/f615b509a2022025f2465341922a2e11427f8822 | |
| 13:55:47 | mriedem | https://opendev.org/x/compute-hyperv/commit/8326d2c43c3e464f0c2e3b58bf57b4ffb9a344e7 | |
| 13:56:18 | lpetrut | those patches look straight forward, should be easy to review/merge | |
| 13:56:51 | lpetrut | well, maybe not the last one :) | |
| 13:56:58 | mriedem | they are at least a specless blueprint just since they are feature additions, but it seems all the feature work on the driver has been out of tree for the last couple of years | |
| 13:57:33 | jaypipes | lpetrut: yes, exactly. | |
| 13:58:32 | mriedem | lpetrut: anyway, just a thought, those first 2 look simple. i'm not looking to generate work, but i'd like to see the in-tree driver close some of these feature gaps that are already in the out of tree driver. | |
| 13:59:39 | lpetrut | mriedem: yep, we can start with a few patches to close the gap a bit. thanks for bringing this up | |
| 14:01:44 | mriedem | bauzas: this is another simple functional recreate test for a stein regression: https://review.opendev.org/#/c/653098/ | |
| 14:05:19 | mriedem | jaypipes: trying to think of ways to maybe debug that oslo.config getattr thing - we could maybe have a DNM patch to oslo.config that counts the number of times we've recursed into that getattr loop and log a traceback to see what's triggering it, then push a nova change that depends-on that oslo.config debug patch? | |
| 14:09:53 | ganso | mriedem: hey Matt! sorry for interrupting the current convo, but just a heads up of https://review.opendev.org/#/c/652153 ... I tested in my env and it did not work. It could be perhaps because I run stable packages instead of directly from source. Did it work in your env? | |
| 14:11:08 | mriedem | ganso: i saw your comment but i haven't yet tried to recreate in a queens devstack | |
| 14:11:23 | mriedem | do you see errors in the source node compute logs? | |
| 14:11:35 | mriedem | did you try rocky? | |
| 14:11:45 | ganso | mriedem: no I did not see any errors. It just did not remove the allocations | |
| 14:11:50 | ganso | mriedem: no I haven't tried rocky | |
| 14:12:33 | mriedem | hmm, i'm not sure what the problem would be since it's calling the same method | |
| 14:13:40 | mriedem | alternatively i could write a functional test to recreate it | |
| 14:14:07 | mriedem | if you can debug it would obviously speed things up | |
| 14:25:21 | jaypipes | mriedem: yeah, the issue is we don't really know where the original call site is for the getattr that is causing the stack exhaustion. if we did know what is the thing that is the base object or thing that is being called there, we could swap out that for either a mock or an actual collections.Mapping etc and see if the issue is in the ovo code, the oslo.cfg code, or something else. | |
| 14:26:20 | jaypipes | mriedem: I'm wondering if there's a way to limit the Python stack recursion size to a smallish number so that the stacktrace would show the entire traceback instead of just the last hundred or so repeated recursive calls to getattr. | |
| 14:26:43 | jaypipes | https://docs.python.org/3/library/sys.html#sys.setrecursionlimit | |
| 14:27:03 | jaypipes | might be worth throwing that in a test, mriedem? ^ | |
| 14:27:11 | jaypipes | just to get the whole traceback. | |
| 14:27:21 | jaypipes | which might give us a better starting point. | |
| 14:32:17 | mriedem | jaypipes: that's what i'm working on - dump the traceback to the logs before we get too far into the recursion | |
| 14:34:12 | jaypipes | mriedem: coo. | |
| 14:34:25 | mriedem | i'll push a nova change which depends-on it and see if we can recreate | |
| 14:35:23 | ganso | mriedem: thanks! | |
| 14:38:01 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: debug config opts infinite recursion https://review.opendev.org/654468 | |
| 15:13:33 | openstackgerrit | Merged openstack/nova stable/pike: Fix incomplete instance data returned after build failure https://review.opendev.org/647916 | |
| 15:15:54 | efried | mriedem: Where are the logs you inserted? | |
| 15:18:46 | efried | I don't see 'em in the job output or subunit results | |
| 15:22:18 | efried | maybe logger config is set higher for oslo? | |
| 15:24:13 | mriedem | efried: it won't dump the logs unless we hit a failure i don't think... | |
| 15:24:16 | mriedem | maybe i need to use prints | |
| 15:24:47 | mriedem | because what fails (TestRPC) isn't where the stack overflow happens i don't think | |
| 15:26:16 | efried | Use prints... and hope the stdout fixture isn't set up | |
| 15:26:16 | mriedem | i don't see any default log level set for oslo.config either https://github.com/openstack/oslo.log/blob/22e8a347c8ba914cefa26df42dde632937259a79/oslo_log/_options.py#L19 | |
| 15:26:51 | efried | so what's the default default if there's no default? | |
| 15:27:04 | mriedem | in unit tests it'd be INFO | |
| 15:27:14 | efried | okay | |
| 15:27:19 | mriedem | which is why we have the OS_DEBUG env var | |
| 16:20:28 | eandersson | Looks good efried | |
| 16:20:39 | eandersson | As for your devstack comment, that was never meant to break | |
| 16:20:44 | eandersson | as you are passing on endpoint = None | |
| 16:20:59 | eandersson | https://review.opendev.org/#/c/653839/ | |
| 16:21:02 | eandersson | This was the actual test | |
| 16:22:02 | efried | eandersson: But that test only reports that our endpoint lookup failed | |
| 16:22:12 | eandersson | Exactly | |
| 16:22:13 | efried | eandersson: ...which is still a valid code path | |
| 16:22:16 | eandersson | Sure | |
| 16:22:21 | efried | like, if you specify a bogus region name or whatever. | |
| 16:22:21 | eandersson | But regions don't actually work | |