| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-02 | |||
| 11:09:46 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Fix backwards compatibility for InstanceNUMACell https://review.openstack.org/396184 | |
| 11:13:04 | johnthetubaguy | sfinucan: I think so, although we should avoid high risk ones till tomorrow | |
| 11:13:37 | johnthetubaguy | sfinucan: we usually try to keep master not too different to stable/ocata so any backports we need are likely to be easier | |
| 11:15:01 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Make 'get_domain' private https://review.openstack.org/417378 | |
| 11:17:57 | johnthetubaguy | mdbooth: ah, good context, in complete cleanup would do it | |
| 11:18:35 | sfinucan | Gerrit's "Add Reviewer" dialog is truly awful | |
| 11:22:19 | openstackgerrit | John Garbutt proposed openstack/nova-specs master: WIP: write up policy ideas for pike https://review.openstack.org/427872 | |
| 11:37:38 | openstackgerrit | John Garbutt proposed openstack/nova-specs master: WIP: write up policy ideas for pike https://review.openstack.org/427872 | |
| 11:40:47 | panda | any hints on what may cause this ? http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/632c7f2/logs/overcloud-novacompute-0/var/log/nova/nova-compute.txt.gz#_2017-02-02_08_13_29_759 | |
| 11:45:55 | kashyap | mdbooth: or johnthetubaguy: When you have a moment, need an opinon | |
| 11:46:10 | kashyap | Where do you think is appropriate for these kinds of tests to live? https://review.openstack.org/#/c/315786/ | |
| 11:46:44 | kashyap | They are trying to add the NMI Watchdog test for a Nova instance in Tempest. But Tempest upstream, rightfully rejects it, as there are no API operations, and it is too Nova-specific | |
| 11:47:03 | johnthetubaguy | kashyap: sounds like we need a nova tempest plugin | |
| 11:47:23 | johnthetubaguy | I would say nova functional tests, but they don't normally need images setup up, AFAIK, I could be wrong | |
| 11:47:34 | johnthetubaguy | kashyap: you really want to be talking to sdague about all that | |
| 11:47:53 | kashyap | johnthetubaguy: Does such a thing exists (or is it exist)? | |
| 11:48:06 | kashyap | johnthetubaguy: Yeah, maybe sdague is still getting ready to wake up | |
| 11:48:23 | johnthetubaguy | kashyap: there are some details here: http://docs.openstack.org/developer/nova/test_strategy.html#types-of-tests | |
| 11:48:30 | johnthetubaguy | but looks like it might need updating | |
| 11:48:51 | kashyap | johnthetubaguy: So, one place you are suggesting is - nova/nova/tests/functional | |
| 11:53:05 | mdbooth | johnthetubaguy: More context on the db_sync thing, looks like we had a puppet job which retried on failure, which included exceeding its timeout. Seems db sync was taking too long and we ended up running 2 simultaneously. | |
| 11:53:14 | mdbooth | So perhaps robustification might be in order. | |
| 11:53:37 | mdbooth | Perhaps the migration should hold an exclusive lock while it runs. | |
| 11:53:44 | johnthetubaguy | mdbooth: it should be all done in a single transaction, for each version, I thought? | |
| 11:54:01 | johnthetubaguy | maybe we never added that stuff in db sync | |
| 11:54:27 | johnthetubaguy | maybe thats not possible... | |
| 11:54:48 | mdbooth | Does msql honour transactions for ddl? | |
| 11:55:00 | johnthetubaguy | I was just thinking the same question, I don't know | |
| 11:55:13 | johnthetubaguy | I had always assume that, but you might be right | |
| 11:55:27 | johnthetubaguy | nothing like an olso.concurrency lock to stop that being an issue, like you say | |
| 11:55:51 | mdbooth | Interwebs say no transactional ddl | |
| 11:56:50 | mdbooth | Think it would have to be a db lock, because we can't be sure where db sync is running. | |
| 11:57:05 | mdbooth | Also, think change would have to be in sqla migrations | |
| 11:58:42 | johnthetubaguy | mdbooth: true, DB lock is safer | |
| 12:10:37 | owalsh | mdbooth: DB lock might be overkill... | |
| 12:11:05 | owalsh | the first db sync succeeded IIRC | |
| 12:11:21 | mdbooth | owalsh: I don't think so. It would only have to used for exclusion of the migration itself. | |
| 12:11:44 | mdbooth | I think it could be pretty simple to implement, tbh. | |
| 12:12:02 | openstackgerrit | Derek Higgins proposed openstack/nova master: Simplify uses of assert_has_calls https://review.openstack.org/428108 | |
| 12:12:16 | derekh | johnthetubaguy: sfinucan ^^ sorry forgot to push it up yesterday | |
| 12:12:26 | mdbooth | When you initially read the current version from the migrate_version table you'd take an exclusive lock on it. | |
| 12:12:52 | mdbooth | Then you'd exclude any other migration from starting until this once completed, but other db operations can continue. | |
| 12:13:15 | sfinucan | derekh: Sweet. Cheers :) Think it can/should wait until Ocata is cut though | |
| 12:13:42 | derekh | sfinucan: sounds fine by me, just letting ye know its there | |
| 12:20:09 | owalsh | mdbooth: hmm the oslo.db tests used to lock - https://review.openstack.org/58830 | |
| 12:20:52 | mdbooth | owalsh: A good reason to use a db lock | |
| 12:21:22 | owalsh | but why protect the tests with a lock instead of the migration? | |
| 12:21:56 | mdbooth | owalsh: I think every test runs the migrations. | |
| 12:22:01 | mdbooth | Unless... optimisation. | |
| 12:22:04 | mdbooth | But mostly. | |
| 12:26:28 | owalsh | forget that, (should have looked at the diff), it's just locking the engine reset | |
| 12:27:20 | mdbooth | owalsh: I've emailed zzzeek for an opinion. cc'd you. | |
| 12:42:21 | lyarwood | hopefully a stupid question, does the gate rebase changes? I keep hitting Bug 1660878 in https://review.openstack.org/#/c/427267/ and wonder if I need to manually rebase it to pick up https://review.openstack.org/#/c/427782/ | |
| 12:42:21 | openstack | bug 1660878 in OpenStack Compute (nova) "test_reboot_deleted_server fails with 409 "Cannot 'reboot' instance while it is in vm_state building"" [High,In progress] https://launchpad.net/bugs/1660878 - Assigned to Matt Riedemann (mriedem) | |
| 12:46:45 | lyarwood | http://logs.openstack.org/67/427267/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/e4e67b3/logs/devstacklog.txt.gz#_2017-02-02_10_50_57_699 - okay, so we merge the change, so I should have https://review.openstack.org/#/c/427782/ | |
| 12:54:24 | openstackgerrit | John Garbutt proposed openstack/nova master: Report reserved_host_disk_mb in GB not KB https://review.openstack.org/428120 | |
| 12:55:46 | johnthetubaguy | bauzas: sfinucan: a nasty placement issue the openstack ansible guys were hitting ^ | |
| 12:55:57 | johnthetubaguy | just want to make sure I am not reading that incorrectly | |
| 13:04:25 | mdbooth | lyarwood: Yep | |
| 13:05:19 | lyarwood | mdbooth: thanks, I'll wait for dansmith and mriedem to come online before I recheck https://review.openstack.org/#/c/427267/ again | |
| 13:08:11 | mdbooth | johnthetubaguy: Do you think you'll get around to https://review.openstack.org/#/c/383859/ before release? Fixes NFS, and makes doing the same for other filesystem-backed drivers trivial. See glusterfs follow-on for eg: https://review.openstack.org/#/c/423402/ | |
| 13:08:13 | gcb | sfinucan, mriedem: As I know we have been preparing for final octata release, Can we still merge python 3 unit tests patches now ? | |
| 13:09:00 | mdbooth | johnthetubaguy: So the NFS change is somewhat complex because it includes infrastructure applicable to all filesystem-backed volume backends. | |
| 13:09:16 | sfinucan | gcb: They're pretty low risk (unit test only) but I think we're only accepting critical bugfixes and doc patches at the moment | |
| 13:09:23 | mdbooth | It's pretty well tested, though. | |
| 13:09:38 | sfinucan | The Pike window will be open tomorrow though | |
| 13:10:02 | mdbooth | It's a bug. Are we passed the bugfix window? | |
| 13:10:10 | mdbooth | past | |
| 13:10:33 | mdbooth | That said, if I can get it into Pike I'll take it and propose for backport. | |
| 13:10:49 | mdbooth | I just want to land the damned thing. | |
| 13:11:13 | gcb | sfinucan: thanks, got it. The only two patches to make all unit tests work on Python 3.5 https://review.openstack.org/#/q/topic:bug/1482633 | |
| 13:11:28 | gcb | it's okay to make them merge into Pike :) | |
| 13:11:51 | sfinucan | gcb: Perfect. I'll take a look tomorrow or Monday :) Will be good to get that all closed off | |
| 13:12:21 | gcb | sfinucan: cool, thanks | |
| 13:18:21 | sfinucan | johnthetubaguy: Took a gawk. Makes sense to me | |
| 13:24:33 | jaypipes | morning folks | |
| 13:24:35 | bauzas | mriedem: FWIW, we have another related problem | |
| 13:24:57 | bauzas | johnthetubaguy: https://bugs.launchpad.net/nova/+bug/1659266 | |
| 13:24:57 | openstack | Launchpad bug 1659266 in OpenStack Compute (nova) "Disk allocation for instance is not good with swap" [High,Confirmed] - Assigned to Tanvir Talukder (tanvirt16) | |
| 13:25:10 | bauzas | but fixing that could be difficult given we already have allocations | |
| 13:25:17 | sfinucan | trinaths: Not sure if you got a response to your question or not. Have you tried building an instance with a NUMA topology but without hugepages? | |
| 13:28:10 | mriedem | gcb: sfinucan: mdbooth: we shouldn't be putting things into the gate or check queue that don't need to be, py3 unit test cleanup included | |
| 13:28:41 | sfinucan | mriedem: Morning. That's what we thought | |
| 13:28:57 | gcb | mriedem: ack | |
| 13:31:32 | mdbooth | lyarwood: I'm just going to grab coffee. Want to talk about swap_volume in a few minutes? | |
| 13:31:43 | lyarwood | mdbooth: sure | |
| 13:31:48 | mriedem | bauzas: is that related to the thing that johnthetubaguy has a patch up for? | |
| 13:32:06 | mriedem | https://review.openstack.org/#/c/428120/ | |
| 13:32:47 | sfinucan | mriedem, bauzas: That's the one, yes | |
| 13:33:04 | bauzas | mriedem: not really but the same problme | |
| 13:33:20 | bauzas | mriedem: honestly, MHO is that we should not use GBs for disk resources but rather bytes | |
| 13:33:37 | bauzas | or KB | |
| 13:33:43 | bauzas | if that's too big | |
| 13:33:54 | bauzas | but, we have lots of ceilings like the one I wrote | |
| 13:34:11 | mriedem | dansmith: melwitt: hmm, lyarwood's patch failed in the gate on the cells inst delete/create race thing http://logs.openstack.org/67/427267/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/e4e67b3/console.html#_2017-02-02_11_48_07_755243 | |
| 13:34:28 | bauzas | mriedem: https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L148 | |
| 13:34:49 | mriedem | https://review.openstack.org/#/c/427782/ | |
| 13:35:07 | mriedem | which merged at 7:34pm | |
| 13:35:29 | mriedem | and lyarwood's change failed at 2017-02-02 11:48:07.755243 | |
| 13:35:51 | lyarwood | mriedem: twice, I think the previous run also hit this a few hours before that | |