Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-04-20
15:40:42 imacdonn mdbooth lyarwood: where do you see that ?
15:40:43 mriedem even that is odd
15:40:52 mdbooth The cinder api makes a synchronous rest call to c-vol, which synchronously makes a rest call to zfssa
15:40:54 dansmith mriedem: not fix the problem, but just warn about the affected uuid(s)
15:40:56 mriedem "someone reported they did this odd thing they shouldn't have, so we're going to handle and log it for them"
15:41:04 mdbooth However, the rest call explicitly isn't synchronous
15:41:05 openstack Launchpad bug 1511466 in OpenStack Compute (nova) "migrate flavor data impossible" [Medium,Confirmed]
15:41:05 zioproto dansmith: I have the proof https://bugs.launchpad.net/nova/+bug/1511466/comments/9
15:41:08 mriedem dansmith: i know, i'm saying it's weird
15:41:10 dansmith mriedem: right now they just get "error, failed, kthx"
15:41:17 zioproto dansmith: somebody wrote on LP to do that, and I did it !
15:41:28 mdbooth The driver waits for the return, but the return it's looking for is ACCEPTED
15:41:31 dansmith zioproto: do you do everything random people tell you to do? nevermind, don't answer that :P
15:41:47 mdbooth Which doesn't indicate the requested change has happened yet, only that it has been requested
15:41:51 mriedem dansmith: so are people doing this because of some latent bug that forced them to do it, but we have since fixed?
15:41:56 dansmith mriedem: well, yeah, but especially if it was advice on a LP bug that multiple people followed, :/
15:42:00 imacdonn mdbooth lyarwood OK, yeah, but doesn't cinder-api wait for the response from cinder-volume before responding to the (cinder) API request?
15:42:23 dansmith mriedem: it doesn't matter, disabling FKs isn't the right workaround for a problem anyway, it's deleting extra, then deleting instance, etc
15:42:28 mdbooth It does, yes, but that response only indicates that zfssa has accepted the request
15:42:32 mriedem dansmith: i agree
15:42:42 mdbooth It doesn't indicate that the request has been processed
15:42:56 mriedem which is why it's odd to put something into code that is for "people worked around this incorrectly, but we're going to patch something in for them"
15:42:58 dansmith mriedem: but regardless, it's out there so I at least want to log/warn and not just fail quietly with no indication about why
15:43:11 imacdonn mdbooth lyarwood: The request to the ZFSSA is another REST API .. it also waits for the operation to complete before returning
15:43:24 mriedem dansmith: where does this handle/log thing go? a migration? an object?
15:43:28 dansmith mriedem: well, I wasn't going to be that nice, I was going to say "Instance Extra (uuid) has no Instance, unable to migrate"
15:43:32 mdbooth imacdonn: It waits for the rest call to return an http response
15:43:35 imacdonn mdbooth lyarwood I know this quite well, because it times out sometimes ;)
15:43:43 dansmith mriedem: in the online migration function that nova-manage is calling
15:43:50 mdbooth But the http resonse doesn't (according to the docs) indicate that it has completed
15:44:06 mriedem dansmith: as you like
15:44:10 mriedem i won't block it
15:44:17 dansmith mriedem: well, I'll put it up and you can decide then
15:44:33 mriedem geddy lee told me that if i choose not to decide i've still made a choice
15:44:35 Tahvok I'm installing nova on ubuntu, and according to the guide I need to create the cell1 cell. In the command they say I need to run it with some sort of id? 109e1d4b-536a-40d0-83c6-5f121b82b650
15:44:37 imacdonn mdbooth lyarwood empirically, I can say that it does wait for the operation to complete .... how else would the status be returned ??
15:44:45 Tahvok What is that id? https://docs.openstack.org/ocata/install-guide-ubuntu/nova-controller-install.html
15:44:48 dansmith mriedem: heh
15:44:51 mdbooth imacdonn: It returns 202 ACCEPTED
15:45:13 mriedem Tahvok: which command? nova-manage cell_v2 create_cell?
15:45:22 Tahvok mriedem: yes
15:45:28 mriedem su -s /bin/sh -c "nova-manage cell_v2 create_cell --name=cell1 --verbose" nova
15:45:36 mriedem that doesn't say anything about providing an id
15:45:39 Tahvok Oh, it's the output
15:45:41 mriedem it outputs an id
15:45:42 mriedem yes
15:45:50 Tahvok Yes, I thought it's part of the command
15:45:55 Tahvok mriedem: thanks!
15:46:16 imacdonn mdbooth: the ZFSSA REST API call ??
15:46:18 mriedem Tahvok: also https://docs.openstack.org/developer/nova/man/nova-manage.html#nova-cells-v2 in case you need more info on the commands
15:46:21 lyarwood mdbooth: sorry back, isn't this a bug within this and any other driver that doesn't terminate_connection synchronously then?
15:46:23 mdbooth imacdonn: Yeah
15:46:38 lyarwood mdbooth: I skimmed over a few target drivers and they appeared to be sync
15:46:39 mdbooth lyarwood: Right, we're discussing zfssa, but other drivers will be different
15:46:54 Tahvok mriedem: is that create_cell command idempotent?
15:47:10 Tahvok Could I run it again, or it will try creating another cell?
15:47:10 imacdonn mdbooth: that doesn't sound right ... looking for a debug log to verify
15:47:12 lyarwood mdbooth: yeah, I think there's still value to adding the lock on our side if we also follow up with cinder to get all drivers in line there
15:48:02 openstack Launchpad bug 1511466 in OpenStack Compute (nova) "migrate flavor data impossible" [Medium,Confirmed]
15:48:02 zioproto dansmith: the bug is still open https://bugs.launchpad.net/nova/+bug/1511466 SOOO I guess any operator with a big deployment that managed to upgrade to liberty probably used this workaround. This means that soon this problem will explode :D
15:48:05 mriedem Tahvok: it's not
15:48:11 mdbooth imacdonn: That's definitely what it returns, and the documented semantics of that are that it's an async call which may not have completed. That doesn't necessarily mean that's what it's doing, of course.
15:48:11 mriedem Tahvok: it would fail with a duplicate entry error
15:48:27 dansmith zioproto: yeah, you know what you need to do to fix your data right?
15:48:33 mriedem Tahvok: "Returns 0 if the cell mapping was successfully created, 1 if the transport url or database connection was missing, and 2 if a cell is already using that transport url and database connection combination."
15:48:38 Tahvok mriedem: that's fine as long as it does not create any new records
15:48:50 mriedem Tahvok: $? would be 2
15:48:57 Tahvok mriedem: thanks a lot!
15:49:02 imacdonn mdbooth: heh, OK ... I'm fairly sure there's nothing in the ZFSSA driver that gets a result of the operation by any means other than the response to the REST request
15:49:13 zioproto dansmith: so you mean that if I delete from the instance_extra table all the records where the uuid is not in instances, then I dont need any patch ?
15:50:04 mdbooth imacdonn: https://docs.oracle.com/cd/E51475_01/html/E52433/makehtml-id-181.html
15:50:24 mdbooth Except it's doing a PUT, not a POST
15:50:42 dansmith zioproto: yeah, that's my expectation.. let me know if it doesn't and we can see if something else is wrong too
15:50:42 zioproto dansmith: I have to rush away from the office... I will ping you tomorrow about this
15:50:46 dansmith zioproto: okay thanks
15:51:01 imacdonn mdbooth: Yeah, so does "successful response" mean "I've completed it"? In practice I believe it does .. so not sure why they're using 202
15:51:58 mdbooth Meh, I wonder if it's reasonable to assert this is sync and make it the cinder driver's problem if it isn't
15:52:03 mdbooth lyarwood: ^^^ ?
15:52:21 imacdonn mdbooth: in any case, I think that nova-compute has to assume that the cinder API call won't return until the work as completed .. if that's not happening, it's either a bug in cinder, or a large other can of worms (need an async mechanism between nova and cinder
15:52:32 lyarwood mdbooth: yeah that's my view
15:52:49 mdbooth lyarwood: Ok
15:53:09 lyarwood mdbooth: but we should really make that assumption known to the cinder folks
15:53:17 lyarwood mdbooth: using this driver as an example
15:53:19 mdbooth Yep
15:53:36 openstackgerrit Dan Smith proposed openstack/nova master: Warn the user about orphaned extra records during keypair migration https://review.openstack.org/458564
15:53:43 dansmith mriedem: this is what I was thinking ^
15:54:04 dansmith mriedem: I'll mark the bug as invalid and also comment on the original bug warning people that they eff'd themselves
15:54:45 zioproto dansmith: with this patch https://review.openstack.org/#/c/458564 I should be able to go on without cleaning the db, right ?
15:54:54 zioproto it is the patch I applied manually, right?
15:55:10 dansmith zioproto: no
15:55:14 dansmith zioproto: it's just a warning
15:55:30 dansmith zioproto: it won't fix the problem, which will make the second keypair migration unable to proceed
15:55:34 dansmith zioproto: you have to clean the database
15:56:04 sfinucan vladikr, sahid: Little cleanup here, if you could take a gawk https://review.openstack.org/#/c/417378/
15:57:13 vladikr sfinucan, sure
15:58:54 openstackgerrit Matt Riedemann proposed openstack/nova master: Deprecate os-hosts API https://review.openstack.org/456504
15:59:01 mriedem alex_xu: ^ fixed that unit test
16:00:38 mdbooth lyarwood: My patch won't fix this, btw, because it needs to be global: https://review.openstack.org/#/c/430213/2/nova/compute/manager.py
16:01:53 mdbooth lyarwood: When using iscsi, I think all connect and disconnect calls must be serialised, where disconnect also include terminate connection
16:02:34 lyarwood mdbooth: global in terms of the compute node right?

Earlier   Later