Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-11-11
17:29:13 mriedem one of the risks of putting a bunch of the driver functionality into a library, which makes things sticky
17:29:25 mriedem i will happily defer to his tonyb-ness on this
17:32:53 openstackgerrit Matt Riedemann proposed openstack/nova master: Add FlavorPayload.description for versioned notifications https://review.openstack.org/514782
17:32:54 openstackgerrit Matt Riedemann proposed openstack/nova master: Add microversion to allow setting flavor description https://review.openstack.org/515213
17:52:04 mriedem efried: is this new? http://logs.openstack.org/73/513473/2/check/legacy-tempest-dsvm-cells/be1a0fa/logs/screen-placement-api.txt.gz#_Nov_10_01_14_05_230382
17:52:11 efried ...
17:52:56 efried mriedem It doesnae look good.
17:53:05 efried looking...
17:54:19 mriedem shit i don't think we index the placement api logs
17:58:32 clarkb we should, though indexing has been funky with the zuul v3 switch (slowly been improving its performance)
17:59:44 efried mriedem ensure_consumer_project_user is only a couple weeks old (https://review.openstack.org/#/c/500410/) but I don't see how the _ensure_project thing could fail unless timing (someone else came along and whacked the entry) or the insert failed.
18:00:51 openstack Launchpad bug 1731668 in OpenStack Compute (nova) "placement: claim allocations fails with IndexError in _ensure_lookup_table_entry" [Undecided,New]
18:00:51 mriedem https://bugs.launchpad.net/nova/+bug/1731668
18:00:54 efried oh. Or external_id ain't in there.
18:01:18 efried naw, cause that should insert it.
18:03:13 mriedem this is during scheduling, so not sure if it's a race or what - it must be a race otherwise we'd have a pretty high failure rate
18:03:23 mriedem 21 hits in 10 days
18:07:20 openstackgerrit Eric Fried proposed openstack/nova master: EXPERIMENTAL: Retry getting project ID https://review.openstack.org/519034
18:07:28 efried mriedem If it's racey, this'll fix it ^
18:07:38 efried But the real deal ought to have a limited number of retries :)
18:11:29 mriedem that seems pretty wrong, i mean, if the insert fails because of a duplicate entry
18:16:16 efried mriedem Depends how racey it is. Clearly the second SELECT failed, or we wouldn't be seeing the error. So we got to the exception clause, inserted a new one, then someone else came in and deleted it before we got to the return.
18:18:09 efried Sorry, let me try that again: The first SELECT returns nada. Then we try to INSERT, but someone had already inserted, so we get the exception. Then before we get a chance to run the second SELECT, someone comes along and deletes the entry.
18:18:25 mriedem true
18:18:54 efried That'd be a pretty wild race to be seeing so frequently.
18:26:04 mriedem in debugging this failed job, it looked like the scheduler was creating allocations twice for the same instance in the same scheduling run, but maybe it was moving the instance or something, i didn't dig into what the test does
18:26:49 mriedem actually this just failed in server create
18:28:26 efried mriedem Where do project IDs ever get deleted?
18:29:31 mriedem tempest cleanup
18:29:38 mriedem but that wouldn't be happening here,
18:29:47 mriedem and the project delete would be via keystone on test teardown in tempest
18:30:09 mriedem when we set allocations in placement, we first remove all of the existing allocations it looks like,
18:30:15 mriedem so i'm wondering if that's related somehow
18:33:33 mriedem the test tries to create 2 servers, the first one gets its allocations created in placement, the 2nd fails to create allocations in placement
18:34:16 mriedem the instances are processed serially and we're only using 1 scheduler worker, so it's weird that something would be racing here
18:35:02 mriedem we should start running 2 scheduler workers in our CI jobs just to tease out concurrency bugs, but that's a separate todo
18:36:23 efried I haven't looked at the test cases yet, but if this external_id is a uuidsentinel that we're repeating in multiple (different, unrelated) test cases, could that be causing collisions?
18:36:29 efried Or does each test case get its own database?
18:42:26 mriedem this is integration testing, so real things
18:42:29 mriedem single db
18:42:48 mriedem i'm not sure what external_id is, is that the project_id in this case?
18:43:03 efried Dunno.
18:43:48 mriedem yeah it is
18:44:23 mriedem we ensure there is an entry in the projects table for that entry before linking it to the consumers table which is linked to the allocations table
18:44:24 mriedem i think
18:44:35 mriedem *entry in the projects table for that project id
18:46:23 mriedem and if we're scheduling 2 instanes with the same project, and the first instance scheduled created allocations successfully, the projects table should already have that entry for that project_id
18:46:39 mriedem so the select() should fine it and we shouldn't even attempt the insert()
18:46:44 mriedem *find it
18:46:49 mriedem damn, brain no work good
18:56:05 openstack Launchpad bug 1730800 in OpenStack Compute (nova) "UnknownConnectionError" [Undecided,New]
18:56:05 mriedem efried: this one is up your alley too https://bugs.launchpad.net/nova/+bug/1730800
18:56:08 mriedem looks like ksa adapter stuff
18:56:26 efried looking...
19:02:11 mriedem battery is gonna die, and so am i, so i'm shutting 'er down. ttyl.
21:42:24 openstackgerrit Merged openstack/nova master: Don't update RT in _allocate_network https://review.openstack.org/513473
#openstack-nova - 2017-11-12
03:25:47 openstackgerrit LIU Yulong proposed openstack/nova master: Enable reset keypair while rebuilding instance https://review.openstack.org/379128
07:14:46 openstackgerrit jichenjc proposed openstack/nova master: [Placement] Add create inventories doc for placement https://review.openstack.org/512215
07:15:49 openstackgerrit jichenjc proposed openstack/nova master: Refactor placement version check https://review.openstack.org/512497
07:16:14 openstackgerrit jichenjc proposed openstack/nova master: Add quiesce and unquiesce in support matrix https://review.openstack.org/482390
07:16:42 openstackgerrit jichenjc proposed openstack/nova master: Add description for resource class creation https://review.openstack.org/508083
08:22:01 openstackgerrit Maciej Kucia proposed openstack/nova master: Add JSON schema validation to PCI whitelist https://review.openstack.org/476098
08:32:13 openstackgerrit Moshe Levi proposed openstack/nova master: Don't overwrite binding-profile https://review.openstack.org/519066
14:51:25 openstackgerrit Alex Xu proposed openstack/nova master: placement: avoid multiple join in _get_all_with_shared https://review.openstack.org/519081
17:37:34 openstackgerrit OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/519127
17:48:24 openstackgerrit OpenStack Proposal Bot proposed openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/519165
18:11:01 efried alex_xu Take a look at the series starting at https://review.openstack.org/#/c/518124/ -- it will make writing RP test cases way easier.
21:07:10 openstackgerrit OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/519127
21:18:28 openstackgerrit OpenStack Proposal Bot proposed openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/519165
#openstack-nova - 2017-11-13
01:29:57 openstackgerrit Yikun Jiang proposed openstack/nova master: Extract SortContext/SortWrapper of instance_list into utils https://review.openstack.org/518871
01:31:43 openstackgerrit Yikun Jiang proposed openstack/nova master: WIP: Add cross cell sort support for get_migrations https://review.openstack.org/517273
01:33:25 openstackgerrit Yikun Jiang proposed openstack/nova master: WIP: Add cross cell sort support for get_migrations https://review.openstack.org/517273
01:57:18 openstackgerrit Zhenyu Zheng proposed openstack/nova master: Mention API behavior change when over quota limit https://review.openstack.org/515631
03:09:11 openstackgerrit Alex Xu proposed openstack/nova master: placement: avoid multiple join in _get_all_with_shared https://review.openstack.org/519081
03:20:35 openstackgerrit Alex Xu proposed openstack/nova master: placement: avoid multiple join in _get_all_with_shared https://review.openstack.org/519081
06:46:34 jianghuaw alex_xu, do you know when OpenStack client will support placement traits?
06:47:04 alex_xu jianghuaw: I don't know that, at least the server side doesn't finish yet
06:47:58 jianghuaw alex_xu, ah. thanks anyway:-)
06:48:06 alex_xu jianghuaw: is there anyone work on the placement support on Openstack Client?
06:49:02 jianghuaw alex_xu, I don't know that.
06:49:39 alex_xu jianghuaw: at least I don't know anyone work on that :)
06:49:57 alex_xu jianghuaw: btw, would you go to the bugsmash next week?
06:50:46 jianghuaw alex_xu, no. I won't attend this time.
06:51:00 jianghuaw I guess you will be there. right?
06:51:10 alex_xu jianghuaw: yea, i will be there
06:52:10 jianghuaw Wish you enjoy the meeting.
07:04:59 openstackgerrit Naichuan Sun proposed openstack/nova master: placement: Modify scheduler and report API to accept parent uuid https://review.openstack.org/519243
07:04:59 openstackgerrit Naichuan Sun proposed openstack/nova master: placement: update client to set parent provider https://review.openstack.org/385693
07:09:46 openstackgerrit Merged openstack/nova master: Refactor test_allocation_candidates https://review.openstack.org/518124
07:09:54 openstackgerrit Merged openstack/nova master: Test alloc candidates with same RC in cn & shared https://review.openstack.org/513149
07:27:47 openstackgerrit Naichuan Sun proposed openstack/nova master: (WIP)placement: Modify scheduler and report API to accept parent uuid https://review.openstack.org/519243
08:36:00 msno hi guyz.
08:37:23 msno whats is the best method to take backup of a boot from volume instance from a openstack instalation.. and restore it in another openstack installation
08:57:46 openstackgerrit Rodolfo Alonso Hernandez proposed openstack/os-vif master: Add abstract OVSDB API https://review.openstack.org/476612
08:58:23 openstackgerrit Rodolfo Alonso Hernandez proposed openstack/os-vif master: Add native implementation OVSDB API https://review.openstack.org/482226
09:00:51 openstackgerrit Rodolfo Alonso Hernandez proposed openstack/os-vif master: Add support for Windows network commands https://review.openstack.org/487405
09:04:13 openstackgerrit Spencer Yu proposed openstack/nova master: Use safer ast.literal_eval instead of eval https://review.openstack.org/519268
09:05:06 openstackgerrit Rodolfo Alonso Hernandez proposed openstack/os-vif master: Change Subnet.dhcp_server to contain multiple IPs https://review.openstack.org/509107
09:14:38 openstackgerrit Zhenyu Zheng proposed openstack/nova master: Add instance action record for attach/detach/swap volumes https://review.openstack.org/517205
09:19:03 openstackgerrit licanwei proposed openstack/nova master: cli 'nova-manage db sync' can't upgrage cell1 https://review.openstack.org/519275

Earlier   Later